Page 1 of 6 12345 ... LastLast
Results 1 to 10 of 54

Thread: Log Parsing tool available

  1. #1
    KWH-PWR#1349Sprt,S Sig#96 scott451's Avatar
    Join Date
    Apr 2009
    Location
    Palo Alto
    Posts
    244

    Log Parsing tool available

    I’ve decided to release my tesla log parsing tool. It’s very basic. It will convert your tesla log file into an excel csv file. There are a lot of interesting values in the log like speed, current, torque, state of charge, etc. There are also a lot of values that are not decoded. I am releasing this tool to enlist your help to decode the remaining values. You can access the parser at the following website. [EDIT 11-15-2010. Website is off line. Please use attached zip file.]


    You will still need to do some data crunching in excel to produce useful plots. As an example I added efficiency as ESS_I * ESS_V / (Line_I * Line_V) to the data collected in the log. Attached is an example plot created in excel from the output of my parser.



    -Scott

    Note: the temp data in the PERM_Daily is wrong. Also the chargeV2 record for 2010 is not supported. (I still need to collect the CAN bus data off a 2010 to figure this one out... )
    Attached Images Attached Images  
    Last edited by scott451; 2010-11-16 at 12:38 AM.

  2. #2
    Head Moderator / Administrator doug's Avatar
    Join Date
    Nov 2006
    Location
    Stanford, California
    Posts
    12,092
    Blog Entries
    10
    pretty cool

  3. #3
    Junior Member Alpine Driver's Avatar
    Join Date
    Mar 2008
    Location
    Austria
    Posts
    91
    well done, Scott, thank you.
    It worked for my 2010 EU signature.
    After download, Excel said "file could not be fully loaded", but that might be because it is more than 65k lines?
    Now I sit here with -a lot- of data. As I am not an Excel expert, I will have to find someone who will make "nice curves" out of that data. What I could read in the plain text is a lot of error messages like "Motor Sensor 1 (or 2) overTemp warning. Torque limited." which might reflect my driving style .
    I also could clarify some charging errror messages I had in the VDS "Charging Error check power source" which in the logfiles much more clearly says "Line underVoltage warning".
    But what I do have extremely often, nearly every day in a dozen is an "isolation resistance warning".
    Everyone else has this "isolation resistance warning"? It is not in the VDS nor in the speedo, so I think it might not be of that influence ... hopefully ...

    best
    Martin

  4. #4
    KWH-PWR#1349Sprt,S Sig#96 scott451's Avatar
    Join Date
    Apr 2009
    Location
    Palo Alto
    Posts
    244
    Quote Originally Posted by Alpine Driver View Post
    well done, Scott, thank you.
    It worked for my 2010 EU signature.
    After download, Excel said "file could not be fully loaded", but that might be because it is more than 65k lines?
    ...
    I also could clarify some charging errror messages I had in the VDS "Charging Error check power source" which in the logfiles much more clearly says "Line underVoltage warning".
    The "file could not be fully loaded" is an Excel 2004 and earlier problem. Excel 2007 doesn't have this limitation. To get around it, look in the spread sheet and find the "utime" that is nearest the event you want to track. Copy that utime into the website "start time" field and re upload the file. The new csv file will start at the new utime. Additonally you can shorten the csv file in wordpad first and then load it into excel.

    Once it is in excel, to get nice curves, data->sort on Column A and then select the data you are interested in and click the chart wizzard.

    The error messages in the log file, will be slightly different than the ones you see on the VDS. The reason is that I use the "verbose error text" not the standard errors. The verbose errors are more descriptive. Collecting the error messages for the parser is a tedious process of analizing the data on the CAN bus and hand typing into my code. I've just about finished some simulation hardware, so this should make it somewhat easier, rather than having to collect the data from my car...

    I should have the 2010 V2 charge data worked out in a few weeks. I need to bring my CAN bus stuff over to a 2010 owner and collect the data, but I've been too swamped at my real job...

    -Scott
    Last edited by scott451; 2010-05-02 at 09:59 PM.

  5. #5
    Quote Originally Posted by Alpine Driver View Post
    But what I do have extremely often, nearly every day in a dozen is an "isolation resistance warning".
    Everyone else has this "isolation resistance warning"? It is not in the VDS nor in the speedo, so I think it might not be of that influence ... hopefully ...
    I saw this once on the VDS. It was when I was running on a race track. I had gotten a number of the motor overtemp warnings that you mention, and once the isolation resistence one, too. I never saw it before or since.

    Maybe what we should take away from this is that your everyday driving is like what I do on the track.

  6. #6
    Member tomsax's Avatar
    Join Date
    Jun 2008
    Location
    Sammamish, WA
    Posts
    743
    Scott,

    Thanks for making your log analysis available to other owners.

    The log files contain a bunch of information including your VIN, your driving second-by-second, your charging, and the GPS coordinates of everywhere you charge, which presumably includes your home location.

    Although there's apparently no GPS data included with the driving records, I suspect that there is enough data that driving routes could be reconstructed by comparing driving patterns (speed, distance and stopping times) with stop lights and traffic flow data in your local area, thus potentially revealing a lot of personal information from those logs.

    What's your data retention and privacy policy for the data that owners upload to your web server?
    Tom Saxton
    Nov 30 2006 to Jun 5 2009 - A long wait for an awesome ride.

  7. #7
    KWH-PWR#1349Sprt,S Sig#96 scott451's Avatar
    Join Date
    Apr 2009
    Location
    Palo Alto
    Posts
    244
    Quote Originally Posted by tomsax View Post
    Scott,

    Thanks for making your log analysis available to other owners.

    The log files contain a bunch of information including your VIN, your driving second-by-second, your charging, and the GPS coordinates of everywhere you charge, which presumably includes your home location.
    Not exactly. The log contains the places where you "limited" the charge. the log does not contain the location of every place you charge. the GPS data is ignored by my parser. The time of charge is not included in the GPS location so it is difficult to correlate to the charge record.

    Although there's apparently no GPS data included with the driving records, I suspect that there is enough data that driving routes could be reconstructed by comparing driving patterns (speed, distance and stopping times) with stop lights and traffic flow data in your local area, thus potentially revealing a lot of personal information from those logs.
    good luck the log does not contain distance travailed per second but speed sampled every second. It also doesn't contain the heading. So you don't know when the GPS location was stored, you don't know how much distance you've travailed in a second, and you don't know which direction you are going. Pretty tough to get any useful data. (e,g where I work, the route I took, etc)

    What's your data retention and privacy policy for the data that owners upload to your web server?
    Log files are used to improve the tool (e.g add more error messages). Parsed data is created and then deleted. uploaded log files are deleted regularly.


    -Scott
    Last edited by scott451; 2010-05-12 at 05:55 PM.

  8. #8
    Member tomsax's Avatar
    Join Date
    Jun 2008
    Location
    Sammamish, WA
    Posts
    743
    Quote Originally Posted by scott451 View Post
    Not exactly. The log contains the places where you "limited" the charge. the log does not contain the location of every place you charge. the GPS data is ignored by my parser. The time of charge is not included in the GPS location so it is difficult to correlate to the charge record.
    I limit charging at home, so my home coordinates are in the log file. I know others do the same.

    good luck the log does not contain distance travailed per second but speed sampled every second. It also doesn't contain the heading. So you don't know when the GPS location was stored, you don't know how much distance you've travailed in a second, and you don't know which direction you are going. Pretty tough to get any useful data. (e,g where I work, the route I took, etc)
    Using speed * time = distance on the once-per-second records yields a reasonable estimate of distance traveled, with maybe a 5% to 10% error. That can be improved using the once-per-ten-minute records that contain the odometer to the 1/10th mile and interpolating. I haven't done the work to see how much accuracy can be obtained, but I can't dismiss that good accuracy is possible.

    You can definitely reconstruct information like this: drove 10.2 miles at low speeds with several stops (city driving), drove 26.3 miles at 75 mph on average (got on the freeway at the exit which is 10.2 miles from home, then got off at the only exit 26.3 miles from the first), then more figuring things out by distances between stop lights. I'm not saying you could always figure things out exactly, but I'm sure someone who wanted to could reconstruct a lot of information.

    I'm sure you're not doing this with the data that owners submit, but it is a privacy issue that I think owners should be aware of when they turn their log files over to Tesla, or anyone else.

    Log files are used to improve the tool (e.g add more error messages). Parsed data is created and then deleted. uploaded log files are deleted regularly
    So you're not keeping any identifiable data for more than a short period of time? That's great to know. I'd recommend stating that on your upload page.

    Despite asking the annoying questions, I think it's great you're sharing your program with the community. Thanks for taking the time to answer my post.
    Tom Saxton
    Nov 30 2006 to Jun 5 2009 - A long wait for an awesome ride.

  9. #9
    KWH-PWR#1349Sprt,S Sig#96 scott451's Avatar
    Join Date
    Apr 2009
    Location
    Palo Alto
    Posts
    244
    Quote Originally Posted by scott451 View Post
    Not exactly. The log contains the places where you "limited" the charge. the log does not contain the location of every place you charge. the GPS data is ignored by my parser. The time of charge is not included in the GPS location so it is difficult to correlate to the charge record.
    Also. The latest version of the [2008, 2010, and sport] firmware encrypts the GPS charging location, vms and user config files, so they can't be read by anyone but tesla.
    Last edited by scott451; 2010-06-12 at 11:08 PM.

  10. #10
    KWH-PWR#1349Sprt,S Sig#96 scott451's Avatar
    Join Date
    Apr 2009
    Location
    Palo Alto
    Posts
    244

    log reader parser Version _v12 online

    I've updated my log parsing tool. The new version includes the following features:

    Correct parsing of ChargeV2 records from 2010 Roadsters (thx to Richard for alowing me to log his 2010)
    Added brick numbers to min temp, max temp, and min v, max V fields.
    Added brake and regen bits to drive1s records (from tomsax PM)
    Added support for parsing older hidden records cause by wraped logs.
    Merged temp and perm logs into one parsing for both 2008 and 2010.
    Added more error text messages based on id# from uploaded logs.

    Known issues:
    For 2010 roadster logs, the Sleep record does not contain the correct the min/max/brick voltages.
    For 2010 roadster logs, the Charge30m record does not contain the motor temperature, so it is set to zero.
    Brick numbers may be off by 1.

    -Scott

    hxxp://logparse.servebbs.com:10376 (change the xx to tt )
    Last edited by scott451; 2010-06-12 at 11:13 PM.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Charging Anxiety Log
    By smorgasbord in forum Model S: Battery & Charging
    Replies: 97
    Last Post: 2012-09-06, 10:51 AM
  2. Can't pull log files
    By Aspirant in forum Roadster: Technical
    Replies: 26
    Last Post: 2012-07-28, 01:53 AM
  3. New member, can't log into my profile
    By deckofficer in forum Site Feedback
    Replies: 15
    Last Post: 2011-12-01, 07:40 AM
  4. Log File Reader
    By SuperB in forum Roadster
    Replies: 16
    Last Post: 2010-02-16, 09:34 PM
  5. Log in yet again
    By vfx in forum Site Feedback
    Replies: 5
    Last Post: 2008-11-06, 02:40 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •