TeslaMS tools for telemetry data visualization

[green1]

I'll preface this by saying that I'm more of a tinkerer than a developer.
I love what's been done with this tools so far, and am just working on setting it up for my own use, I see it has a web-server on it to feed the visualization pages, but if I'm running this on a machine that's already got apache running, can I not just use that instead? (I'm guessing I'm missing something blindingly obvious here)

 

[FlasherZ]

Hey Hans,

Well, it seems my experimentation with a narrower nap time frame has created too many requests.

HTTP/1.1 429
[...]

You have been blocked for 86400 for making too many requests!

As it turns out, this makes the streaming daemon puke and exit:

Error parsing response to oauth token request
10 Jan 11:30:02 - Error: unable to parse vehicle data, login failed

...and then it exits.

Perhaps on a 429 it should just delay and re-try to fetch a token in 15 minutes or so?

(The weird thing here is that the remote app works on my phone yet... they expired the tokens from my collection host, even my manual curl tokens, but left the mobile ones active.)

 

[hans]

I'll preface this by saying that I'm more of a tinkerer than a developer.
I love what's been done with this tools so far, and am just working on setting it up for my own use, I see it has a web-server on it to feed the visualization pages, but if I'm running this on a machine that's already got apache running, can I not just use that instead? (I'm guessing I'm missing something blindingly obvious here)

Most of the web pages are dynamically generated by the code so it's not so easy to integrate with an existing web server that may or may not support node.js and server-side javascript. Packaging the whole thing together with no external integration needed avoids a lot of problems and unnecessary debugging.

The web server that's included in the visualize app can be made to run an any port so that it doesn't conflict with any other pre-existing HTTP servers also running on the same system. Just specify the -p (or --port) flag and give it an unused port. The default is to use 8766. Alternatively you can set an environment variable called $PORT to override the default port (useful for Heroku and other Cloud deployments).

- - - Updated - - -

Hey Hans,

Well, it seems my experimentation with a narrower nap time frame has created too many requests.



As it turns out, this makes the streaming daemon puke and exit:



...and then it exits.

Perhaps on a 429 it should just delay and re-try to fetch a token in 15 minutes or so?

(The weird thing here is that the remote app works on my phone yet... they expired the tokens from my collection host, even my manual curl tokens, but left the mobile ones active.)

Ouch. Sorry about that. I have a request counter that is supposed to prevent too many requests. There is even a "requests per minute" --rpm parameter to control that too. I will take a look but perhaps I missed incrementing the request counter somewhere.

 

[green1]

Most of the web pages are dynamically generated by the code so it's not so easy to integrate with an existing web server that may or may not support node.js and server-side javascript. Packaging the whole thing together with no external integration needed avoids a lot of problems and unnecessary debugging.

The web server that's included in the visualize app can be made to run an any port so that it doesn't conflict with any other pre-existing HTTP servers also running on the same system. Just specify the -p (or --port) flag and give it an unused port. The default is to use 8766. Alternatively you can set an environment variable called $PORT to override the default port (useful for Heroku and other Cloud deployments).

My main concern was about security. If I'm putting it on a machine with a public IP, I'd rather that it's not available to just anyone who happens by. With integration in to an existing webserver I could use the existing authentication methods.

 

[hans]

My main concern was about security. If I'm putting it on a machine with a public IP, I'd rather that it's not available to just anyone who happens by. With integration in to an existing webserver I could use the existing authentication methods.

You can configure basic login/password authentication for visualize by adding the following to your config.json file.

[COLOR=#BFBFBF][FONT=Menlo] 
[/FONT][/COLOR][FONT=Menlo]"visualize":[/FONT]
[FONT=Menlo][[/FONT]
[FONT=Menlo]    { "id": 1, "username": "dirk", "password": "secret" },[/FONT]
[FONT=Menlo]    { "id": 2, "username": "bob", "password": "different" }[/FONT]
[FONT=Menlo]][/FONT][COLOR=#F5F5F5][FONT=Menlo][COLOR=#bfbfbf]
[/COLOR][/FONT][/COLOR]

In addition you can always run a reverse proxy like NGINX or haproxy in front of visualize (and any other web server). This allows for a unified layer to add SSL and any other kind of authentication or access control you want.

 

[green1]

You can configure basic login/password authentication for visualize by adding the following to your config.json file.

[COLOR=#BFBFBF][FONT=Menlo] 
[/FONT][/COLOR][FONT=Menlo]"visualize":[/FONT]
[FONT=Menlo][[/FONT]
[FONT=Menlo]    { "id": 1, "username": "dirk", "password": "secret" },[/FONT]
[FONT=Menlo]    { "id": 2, "username": "bob", "password": "different" }[/FONT]
[FONT=Menlo]][/FONT][COLOR=#F5F5F5][FONT=Menlo][COLOR=#bfbfbf]
[/COLOR][/FONT][/COLOR]

In addition you can always run a reverse proxy like NGINX or haproxy in front of visualize (and any other web server). This allows for a unified layer to add SSL and any other kind of authentication or access control you want.

Thanks! that's exactly what I was looking for! (I like the reverse proxy idea, but one step at a time. As I said, more of a tinkerer than a developer...)

 

[Mario Kadastik]

Seems the tools are now working again or at least I got them running on my laptop for now. The question is for multi-car accounts, I can see the cars with teslacmd -a, but how can I specify which car to show data for when using for example -i or other options? And if I want to use the streaming API, how could I make it record the streaming data for cars 1,2,3,5 or for all cars or just for one specific car, but to have various DB's to put the data to. I'd love to move all our cars to one account for easier app usage later on and that might mean we'll have 7 cars on one account or we might put them all to separate accounts and I'll stream them separately, but would be nice to know how the tools handle it?

Edit: nevermind, -O #

 

[green1]

You can configure basic login/password authentication for visualize by adding the following to your config.json file.

[COLOR=#BFBFBF][FONT=Menlo] 
[/FONT][/COLOR][FONT=Menlo]"visualize":[/FONT]
[FONT=Menlo][[/FONT]
[FONT=Menlo]    { "id": 1, "username": "dirk", "password": "secret" },[/FONT]
[FONT=Menlo]    { "id": 2, "username": "bob", "password": "different" }[/FONT]
[FONT=Menlo]][/FONT][COLOR=#F5F5F5][FONT=Menlo][COLOR=#bfbfbf]
[/COLOR][/FONT][/COLOR]

Ok, I seem to be missing something. I did as you said, but when I go to the webserver it doesn't ask for a username or password...

Here's my config.json everything seems to work fine, except no authentication.

}
    "username": "____________",
    "password": "____________"
}


"visualize":
[
    { "id": 1, "username": "_____", "password": "______" },
]

EDIT:
After reviewing the thread some more I modified my config.json file as follows:

{"username": "_______________",
"password": "_______________",
"visualize":
[
    { "id": 1, "username": "_______", "password": "______" } 
]
}

This got rid of the error in the log file about no "visualize property" and authentication being off. Now the log simply shows:

Using MongoDB URI: mongodb://127.0.0.1:27017//home/tesla/tesla
Server running on port 8766

But I'm still not asked for any login details when I go to that page.

 

[hans]

After reviewing the thread some more I modified my config.json file as follows:

{"username": "_______________",
"password": "_______________",
"visualize":
[
    { "id": 1, "username": "_______", "password": "______" } 
]
}

This got rid of the error in the log file about no "visualize property" and authentication being off. Now the log simply shows:

Using MongoDB URI: mongodb://127.0.0.1:27017//home/tesla/tesla
Server running on port 8766

But I'm still not asked for any login details when I go to that page.

Seems like you have the JSON syntax right. Are you sure you are editing the config.json that the app is actually using and not some other copy of the file?
Is it the file in in the .teslams folder under your home directory (i.e. ~/.teslams/config.json) or the one in the directory where the example apps where installed?
I suppose if you no longer get the error it must be the one you are editing. I don't use the password feature myself so tomorrow I will set this up and try and duplicate the issue.
I just got an update to the 2.9.x firmware so I can test a few more things as well now.

I also have a prototype version of streaming.js that publishes mqtt messages and another option that now supports Amazon AWS IoT, DynamoDB for storing data, and Lambda for writing rules on the streaming data.

Next up is a web app for realtime display or steaming data and perhaps a raspberry pi touchscreen app for building a home Tesla monitor and control app.

 

[FlasherZ]

Seems like you have the JSON syntax right. Are you sure you are editing the config.json that the app is actually using and not some other copy of the file?
Is it the file in in the .teslams folder under your home directory (i.e. ~/.teslams/config.json) or the one in the directory where the example apps where installed?
I suppose if you no longer get the error it must be the one you are editing. I don't use the password feature myself so tomorrow I will set this up and try and duplicate the issue.
I just got an update to the 2.9.x firmware so I can test a few more things as well now.

I also have a prototype version of streaming.js that publishes mqtt messages and another option that now supports Amazon AWS IoT, DynamoDB for storing data, and Lambda for writing rules on the streaming data.

Next up is a web app for realtime display or steaming data and perhaps a raspberry pi touchscreen app for building a home Tesla monitor and control app.

I use the feature.

You're missing the "webusers" object, and because of that, you need braces for the "visualize" object.

Here's my (santized) config.json:

{
 "username": "[email protected]",
 "password": "yeahnosorry",
 "visualize": {
  "baseUrl": "/tesla",
  "webusers": [
    { "id": 1, "username": "flasherz", "password": "dontthinkso" }
  ]
 }
}

If you don't need the baseUrl, just delete that line and you'll be fine.

 

[hans]

I use the feature.

You're missing the "webusers" object, and because of that, you need braces for the "visualize" object.

I will update the README since its not right. Thanks.

 

[green1]

You're missing the "webusers" object, and because of that, you need braces for the "visualize" object.

perfect! that did it, thanks!

- - - Updated - - -

I also have a prototype version of streaming.js that publishes mqtt messages and another option that now supports Amazon AWS IoT, DynamoDB for storing data, and Lambda for writing rules on the streaming data.
Next up is a web app for realtime display or steaming data and perhaps a raspberry pi touchscreen app for building a home Tesla monitor and control app.

Exciting times ahead!
I'm basically interested in anything I can add to my web server to allow control of, and data collection from, my car. My ideal would be a full on replacement of visible tesla, but on my web server and available from any device with a browser instead of just my PC.
I can tinker decently, and often modify existing stuff a little bit, but I'm just no good at writing these things myself. Thank you to everyone who has been doing the hard part for me so far!

 

[FlasherZ]

I will update the README since its not right. Thanks.

No problem. It still doesn't act correctly in some contexts and I haven't tried to track it down. Basically, I can log in and use the visualization for a while - but if I come back in a half-hour, the page doesn't load right. It's probably something to do with caching and authentication, but even a shift-reload won't fix it. I didn't try ctrl-reload. I'll try to find the failure scenario when I get some time.

 

[green1]

Noting a couple things in the visualize.js file,

First, there's a couple things that each user needs to change, maybe these should be loaded from config.json? or at lest emphasized in the readme. These include the google maps API, and the local secret.

I'm also discovering that the options decoder isn't working right, I'm playing with that now and I'll see what I can do.

edit:

ok, I was finding that the options decoder wasn't working right for my P85+ so I fixed it, I also added P85D and 70 to the list (I'm sure there's a code for awd somewhere, but I don't know what it is

var baseText = {
    RENA: "North American",
    REEU: "European",
    TM02: "Signature",
    PF00: "S",
    PBT85: "P85",
    PX01: "PLUS",
    BT60: "60",
    BT40: "40",
    P85D: "P85D",
    BT70: "70"
};

 

[hans]

I'm also discovering that the options decoder isn't working right, I'm playing with that now and I'll see what I can do.

The options and firmware decoders haven't been updated in a long time so it would be great if someone could bring them up to date. Either make a Github pull request (preferred) or send me the changes and I can review them and check them into the master codebase.

 

[green1]

The options and firmware decoders haven't been updated in a long time so it would be great if someone could bring them up to date. Either make a Github pull request (preferred) or send me the changes and I can review them and check them into the master codebase.

Ok, I've never tried to work with Github before, but I think I created a pull request to merge in the fixes to the model decoder. If you can confirm that I did that part right, then I'll go ahead and work on the options, and firmware versions, and who knows what else I might be able to do!

(I should mention on github I'm ve6rah not green1)

-- EDIT --

I should mention that I've discovered another interesting issue with this. Seems the web server integrated in to visualize.js doesn't listen on IPv6 any ideas how to fix that?

 

[ohmman]

For those of you using a Mac to run the streaming service, I've created the following launchctl file:

Create

~/Library/LaunchAgents/com.hjespers.teslams.streaming.plist

with content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
		<string>com.hjespers.teslams.streaming</string>
	<key>KeepAlive</key>
		<true/>	
	<key>RunAtLoad</key>
		<true/>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/local/bin/node</string>
		<string>/usr/local/bin/streaming</string>
		<string>-z</string>
		<string>--db</string>
		<string>teslams</string>
	</array>
</dict>
</plist>

Note that my MongoDB name is teslams, so you'll want to change that to whatever you use. I also have the -z sleep option being passed. You'll want to check paths, etc.

Start with:

launchctl load ~/Library/LaunchAgents/com.hjespers.teslams.streaming.plist

Stop with:

launchctl unload ~/Library/LaunchAgents/com.hjespers.teslams.streaming.plist

Should automatically respawn after dying, and it should start on boot.

 

[green1]

There were some updates recently, and I tried to update the version I'm running, and something obviously didn't go according to plan... ideas?

This is what shows up in visualize.log and the service doesn't seem to be running:

/usr/local/lib/node_modules/teslams/node_modules/mongodb/lib/url_parser.js:128
      throw new Error('More than 1 database name in URL');
            ^
Error: More than 1 database name in URL
    at module.exports (/usr/local/lib/node_modules/teslams/node_modules/mongodb/lib/url_parser.js:128:13)
    at connect (/usr/local/lib/node_modules/teslams/node_modules/mongodb/lib/mongo_client.js:125:16)
    at Function.MongoClient.connect (/usr/local/lib/node_modules/teslams/node_modules/mongodb/lib/mongo_client.js:109:3)
    at Function.<anonymous> (/usr/local/lib/node_modules/teslams/examples/visualize/visualize.js:253:17)
    at Function.exports.namespace (/usr/local/lib/node_modules/teslams/node_modules/express-namespace/index.js:31:6)
    at Object.<anonymous> (/usr/local/lib/node_modules/teslams/examples/visualize/visualize.js:177:5)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
Using MongoDB URI: mongodb://127.0.0.1:27017//home/tesla/tesla

 

[hans]

Using MongoDB URI: mongodb://127.0.0.1:27017//home/tesla/tesla
[/code]

What database name did you mean to use and what was the command line you used to start this? There were a small number of changes to visualize.js so it should be hard to find what broke.

 

[green1]

What database name did you mean to use and what was the command line you used to start this? There were a small number of changes to visualize.js so it should be hard to find what broke.

database is /home/tesla/tesla and was working fine on the last version (as in the version from a week ago)

sudo -u tesla /usr/local/bin/node /usr/local/lib/node_modules/teslams/examples/visualize/visualize.js --db /home/tesla/tesla >> /var/log/visualize.log 2>&1

I think I might see it. I set the database in my command line as /home/tesla/tesla but in the log it shows mongodb://127.0.0.1:27017//home/tesla/tesla (note the double slash after the port number) Is that second slash supposed to be there? or did something change there that's now confusing matters?

EDIT: nevermind, I took the slash out of my command line and it didn't help anything. so that's not what's causing the issue.... other ideas?