23 June 2011

Reading a text or CSV file using the Processing language

In a previous post, I showed you how to export data to a text file. Now I will show you how to import it back into your Processing program. This will come in handy later on.

This is what my data looks like in the text file:

There are many ways to import text from a text file, and there are many ways to store the data within your code after you have imported it. Feel free to make mention of your method in the comments, however, this one definitely works, and is doing what I want it to do.

I have decided to store each line within an ArrayList.
The first column is stored in another ArrayList by splitting the data into bits using splitTokens.
Most of the file reading code was taken from this Java site, and seems to work quite fine with the Processing programming language. I have taken bits and pieces of code from various places, and added my own flavour.

Here is a snippet of the data after import.


You can see that I have separated the two ArrayLists using dots "....."



Processing Code:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
/*==========================================================
Project: Read data from Text file into Processing
Author: ScottC
Created: 23rd Jun 2011 (Updated 26th Aug 2014)
Description: Use processing to read a text file and populate an ArrayList.
             The ArrayList is then printed to the debug window.
             
Processing version tested: 2.2.1
References: This was made possible using bits and pieces from these sites
         http://www.kodejava.org/examples/28.html
         http://processing.org/reference/ArrayList.html
         http://processing.org/reference/splitTokens_.html
         
===========================================================  */
import java.io.FileReader;
import java.io.FileNotFoundException;

ArrayList sensorData;
ArrayList columnOne;

void setup(){
  sensorData=new ArrayList();
  columnOne=new ArrayList();
  readData("C:/SensorData/mySensorData.txt");
}

void readData(String myFileName){
  
  File file=new File(myFileName);
  BufferedReader br=null;
  
  try{
    br=new BufferedReader(new FileReader(file));
    String text=null;
    
    while((text=br.readLine())!=null){
      String [] subtext = splitTokens(text,",");
      columnOne.add(int(subtext[0]));
      sensorData.add(text);
    }
  }catch(FileNotFoundException e){
    e.printStackTrace();
  }catch(IOException e){
    e.printStackTrace();
  }finally{
    try {
      if (br != null){
        br.close();
      }
    } catch (IOException e) {
      e.printStackTrace();
    }
  }
  for (int i=0; i<sensorData.size()-1; i++){
    print(columnOne.get(i) + ".....");
    println(sensorData.get(i));
  }
}

 Processing Code Explained


void setup(): this creates 2 new ArrayLists to hold the data, and then calls the readData function. The readData() function needs the name of the file that you want to analyse, eg. C:/mySensorData.txt


br=new BufferedReader(new FileReader(file));
This just sets up the file that you will be reading, which then allows you to use a "while-loop" to read one line at a time using br.readLine().


String [] subtext = splitTokens(text,",");
This splits each line into bits, using a "," as a separator, and puts it into the subtext array.


columnOne.add(int(subtext[0]));
This shows how you can extract the first number or column from each line.
If you wanted the second number or column, you would use subtext[1] instead.


sensorData.add(text);
This shows how you can extract the ENTIRE line.


  }catch(FileNotFoundException e){
    e.printStackTrace();
  }catch(IOException e){
    e.printStackTrace();

This is just error handling related to file reading.


br.close();
This closes the file.


for (int i=0; i<sensorData.size()-1; i++){
    print(columnOne.get(i) + ".....");
    println(sensorData.get(i));
  }

This just prints the data to your screen so that you can see if the whole process has been successful.




Update : If you want to read values from a text file and then send these values to the Arduino - read this blog entry.







39 comments:

  1. Is it possible to read live data from a text file?
    I want to read a text file that autoupdates. It has numbers inside, but I want to convert those to variables and send them over to Arduino. Then the Arduino would use those variables.
    Thank you!

    Razvan.

    ReplyDelete
  2. Good question Razvan: I cannot see why not. But not sure if it is the best way to do it. Perhaps you could get the processing script to grab the data from the original source rather than double handling. But once the Processing script has the variables, you can send these to the Arduino via a serial command, and you should be able to get the Arduino to respond appropriately.

    ReplyDelete
  3. Nice post!
    I am a beginner in processing. I am trying to read a text file from SD card on Arduino. The space separated text file has 4 fields (i.e. 4 columns) of sensor data. I am trying to compare the data with a live sensor connected to arduino.

    Well, if you can please explain the post a little bit more.
    (You can do that by commenting each line of the code).

    ReplyDelete
  4. Hi, could you tell me how to convert sensor.Data.get(i) to integer, thx

    ReplyDelete
  5. Sarvesh:
    I would encourage you to read the websites provided:
    http://www.kodejava.org/examples/28.html
    http://processing.org/reference/ArrayList.html
    http://processing.org/reference/splitTokens_.html

    and then come back and ask about the part of the code you don't understand.

    ReplyDelete
  6. This is one way to convert a string to an int:

    String myString = "16";
    int myInt = int(myString);

    or

    int myInt = Integer.parseInt(myString);


    Please note that sensorData.get(i) is not a number, it is a set of numbers separated by commas.
    If you want the individual numbers, then you will have to set up more ArrayLists (eg. ColumnTwo, ColumnThree etc etc), and grab them in the same way that I did for the ColumnOne Arraylist. See the "Processing Code Explained" section above.

    ReplyDelete
  7. There's no library to use FILE?

    ReplyDelete
  8. I am assuming that you are talking about the processing language ?

    ReplyDelete
  9. so once you are able to read the array through processing .. u can send it to arduino??? by any chance do you happen to have an example for that one

    ReplyDelete
    Replies
    1. Have a look at this post: http://arduinobasics.blogspot.com/2012/05/reading-from-text-file-and-sending-to.html

      Delete
  10. I'm trying to get the arduino to read a single number inside a txt file that is constantly updated. Do you know the code for doing this.

    ReplyDelete
    Replies
    1. Dear Anonymous,
      Have you tried asking in the Arduino forums?

      Delete
    2. Have a look at this post: http://arduinobasics.blogspot.com/2012/05/reading-from-text-file-and-sending-to.html

      Delete
  11. Hello I'm trying to send a text file from my computer to Arduino UNO using Arduino software and also send that text file from Arduino UNO to another Arduino UNO through wireless using two Xbee S2. Can you tell me how to do it. can I do it using Arduino Software or I have to use Processing Language.

    My text file contains three columns, and three headings for each and the data under the headings is almost 7500 values for each column.

    Can you please help me how to code it.
    thanks

    ReplyDelete
    Replies
    1. I only know how to do this using the Processing Language. But based on what you describe, it should be possible to do what you want - using the Processing language.
      You don't have to use Processing, you could use a Java, VB.NET, Python (choose a language of your choice) program to send the information to Arduino using a Serial Interface.
      Once your 1st Arduino gets the information, it should not be hard to transmit it to the 2nd Arduino using an XBee.

      See the following post for some tips on how to send a text file to an Arduino:
      http://arduinobasics.blogspot.com.au/2012/05/reading-from-text-file-and-sending-to.html

      Arduino Forums are also a useful way of getting help with coding etc.
      http://www.arduino.cc/forum/

      Much quicker than I could ever do :)

      Delete
    2. Thanks a lot for the response, I wanted to know if the python is executable in Arduino software, else I have to use Python 2.7 software to run it?

      I can use Processing language as well if others are time taking, I can be adaptable to any language. So can you please suggest me which is good to go to start away my project.

      Thanks for the help :)

      Delete
    3. You would run the Python/JAVA/C++/VB.NET/Processing program on your computer. The USB cable attached to the Arduino will be used as a virtual serial port. Your executable on your PC will communicate and transfer information from your hard drive to the Arduino via the USB cable.

      The Arduino can be programmed to perform various tasks using the data being sent from the computer. To do this, you will need to use the Arduino IDE.

      I have provided you with a link that will get you started on the right track (see previous comment). So I would suggest a Processing/Arduino software combination. But you can use any other programming language on your computer. However you should use the Arduino IDE to program the Arduino.

      The Processing language is well documented and is very similar to C++ and Java - the Processing IDE can be downloaded for free from : http://processing.org/download/

      The Arduino IDE can be found here: http://arduino.cc/en/Main/Software

      Like I said before, the Arduino Forums are very useful for getting the answers you need.

      Delete
    4. I have actually used Arduino to communicate with othr Arduino using the Arduino IDE. and I also have installed Processing IDE. I'm unable to link between these two, i mean I copied the Arduino Libraries to Processing libraries but still I'm unable to use Processing IDE to communicate with my Arduino

      Delete
    5. How to I make processing IDE to communicate with my Arduino? Can you please tell me.

      Delete
    6. Processing runs on your computer. The Arduino Program runs on the Arduino.
      You cannot run processing on your Arduino.
      Processing runs on your PC, communicates with Arduino through USB.
      Follow the example I provided in the link. It will show that you use Serial commands to communicate with the Arduino.

      Delete
    7. the text file which I want to send is 200KB. Can Arduino store this size? or else Shall I use the SD card for sending the data?

      Delete
    8. It depends on the type of Arduino you have:
      Arduino Uno: http://arduino.cc/en/Main/ArduinoBoardUno
      Arduino Leonardo: http://arduino.cc/en/Main/ArduinoBoardLeonardo
      Arduino Mega 2560: http://arduino.cc/en/Main/ArduinoBoardMega2560
      Arduino Mega ADK:http://arduino.cc/en/Main/ArduinoBoardADK

      Unless you have an Arduino Mega, you will need to use an SD card to store the data. However, it depends on what you are trying to do with the data. What will the Arduino do with the data after receiving it?

      Delete
    9. It will just send it to the computer. I'm using Arduino just for wireless transfer. I reduced the file to 105KB but still im not able to figure it out how to store the data into the Arduino and also how to send from one Arduino to another Arduino.

      I'm using Arduino Uno board with Xbee Series 2 for both the Arduino.

      Can you please help me.

      Delete
    10. Dear Anonymous,

      1. I don't think you need to send all of the data at once. Send one line or one value at a time, and pass it on. You can easily use an array, or a variable to do this. No need to use valuable Arduino memory for this task.

      2. You are not trying to send whole numbers, you are trying to send values like: 0.016, -0.305, -0.195, And while it is possible to transfer this data through an Arduino, it will take a bit more effort than the examples I have provided so far.

      3. I would recommend that if you have a value of 0.4, that you store this value as 0.400. This will make it much easier to write the code necessary to transfer the data.

      4. I would also recommend that you delimit the values with a comma or some other type of separator. And perhaps include a line delimiter such as a * or some other symbol at the end of each line.

      5. It looks like you are trying to jump into more advanced stuff, without learning the basics. Rather than trying to get someone to do all the hard work for you, maybe try and learn the simple stuff first, and once you understand how it works, you will then be able to put it all together.

      6. Yes - I can help you. But I am taking the long road. Feel free to join me - I will post a link soon to a progressive tutorial that will hopefully meet your needs - and if you take the time to follow through the examples I provide you, it will all become clear (I hope).

      Delete
    11. Dear Anonymous:

      You can start here:

      http://arduinobasics.blogspot.com/2012/07/arduino-basics-simple-arduino-serial.html

      Delete
    12. Dear ScottC,

      I'm ready to learn, I was asking for the basic examples to start with and yes I got it how to write the program using array. I'm looking forward for the examples so that I can write my own code. Please try to provide me some examples even I'll find out meanwhile.

      Thanks

      Delete
    13. Does this require Processing IDE?? or only Arduino will do?

      Delete
    14. I don't know how to get the Arduino to read a computer's hard drive. However, I do know how to get a program that is running on your computer to read text files (as per comments above). Therefore I would use a Processing script to read the text file. Processing is also capable of communicating with the Arduino through the serial port. This will become clear when you go through the progressive tutorial which can be found here:
      http://arduinobasics.blogspot.com/2012/07/arduino-basics-simple-arduino-serial.html

      I haven't got to the bit where the Processing is used, but please be patient, I will get there. We need to learn to walk before we can run.

      Delete
  12. Hi in my datalogger i used the string datasring and had my values in csv format in excel. the problem is that it only increment to 7 or 10 data. and it will restart. I know there's something wrong with my program. So im planning to use this code. I also wish that my logger will only count if theres someone who pass that sensor. I use digital output. When high it is one when not low and i don't want to log all the 0(low) data in my logger.

    iggy

    ReplyDelete
    Replies
    1. Hi Iggy,

      Not sure if this is a question or a statement. Please clarify if this is a question??

      Delete
  13. Hi there

    I am new to arduino, is there any way to do the same thing above but using the ardunio wifi shield V2.1? how would i communicate between the two using processing?

    ReplyDelete
    Replies
    1. Hi Anonymous,

      I don't have a Wifi shield, so am not sure how you would do it. I would ask the Arduino/Processing forums, they are much better at answering these kinds of questions.

      I would start by trying to communicate with one Arduino via wifi.
      I assume that this could be done via a Serial command???

      Delete
  14. Thank you for your quick response , yes i do have it communicating using putty just trying to figure out how to use it with processing.

    Thanks again i will try asking them

    ReplyDelete
  15. I have a Excel file with 3 columns(Column A,B and C).. I wanna read the excel file in arduino Uno board and find maximum of each column (say MaxA for Column A , MaxB for Column B and MaxC for Column C)..

    Regards
    Vignesh

    ReplyDelete
    Replies
    1. Why not find the maximum in excel, and save the maximums to a file that the Arduino can read?

      Delete
  16. Hello, I am getting a new window but it is empty and there is nothing coming in it. Program is showing no errors, no warnings. I don't know why it is doing like this. I am reading a text file which is having "help me" text stored.

    ReplyDelete
  17. Maybe with createReader is easier... https://processing.org/reference/createReader_.html

    ReplyDelete
    Replies
    1. Processing has been updated since I wrote this tutorial - createReader is worth looking at.

      Delete

Feel free to leave a comment about this tutorial below.
Any questions about your particular project should be asked in the ArduinoBasics forum.

Comments are moderated due to large amount of spam.