Logging for data analytics

I’m a programmer/data scientist/hacker/cybersecurity expert/electronics engineering degree holder but not user and that’s a bad combo for thinking out of the box. I’ve been thinking about making a data logging and data analytics module for the CNCs. Ideally ran on Linux PC or a pi.

My idea is to integrate sensors into the machine. For example thermistors on all the motors, rpm sensor on the spindles, voltage/amperage sensors on the power supply and motor drivers. Maybe more sensors on the boards. Anyway my idea is to sensor all the things and log everything.

Convert and log all this data, and input it into a SIEM or similar. I have most of my experience in splunk/elastic search and jupyter notebook but I’m not sure they would be appropriate for this use case. Perhaps an IoT tool I don’t know about but could research.

Anyway the idea is to use this tool to monitor our workflows, create more effective and efficient cuts, get an idea on the health of our machines, early warning for malfunction or even automated emergency stops.

Just an idea I thought I’d throw out there for discussion. I know there’s a few programmers and such on this board who could maybe provide some insight.

1 Like

You’ve already signed up for the Trans-Human Trials, haven’t you?

1 Like

Was I not supposed to?

I was going to ask what the process was… I’m getting tired of the current parts wearing out.

The most interesting would be to put encoders on the steppers I think, and log the actual position of the steppers. I think that the difference between the actual and desired position can be a very good estimate for the load on the end mill, which is a very important variable if you want to make very accurate parts.

Or, put an accelerometer on the center assembly. Measure the tilt of the end mill using the accelerometer (not sure if that works very well). Or measure the vibration while cutting. Too much vibration / chatter is not good and might break your end mill and indicates that you might want to refine your CAM.

Yes that is a great idea!

It’s a bit more challenging but I think a strain gauge would be awesome. Then a machine learning model that predicts tool load and adjusts feed rate accordingly. Because closed loop is cheating and anyway would be less data science-y.

1 Like

That will be data collecting just for data collecting.
But anyway if you like to do this for fun you can start with a set of cheap i2c sensors that connected to esp8266 or esp32 modules. Modules flashed by esphome firmware, that over wifi pushes data to raspberry pi powered by hassos. On hassos you can setup influxdb plugin and if you like graphana plugin. You need a good microsd card. That’s it.

Why hassos? The rest makes good sense. I don’t see the need for any of the other benefits of hassos in this instance unless I’m missing something.

And I don’t believe it would just be data collection for no purpose. I think there could be beneficial uses. I envision something like the OBD2 system in our cars. Sure 90% of the time it’s useless, but modern day engine diagnosis is simplified significantly by modern sensors and data logging.

I’m sure someone right now is thinking that I am wrong with engine diagnosis, and they think a carbureted pushrod engine is easier to diagnose and I don’t want it to distract from the current conversation so I’m going to address it preemptively. If you believe this you’re wrong. The reason your wrong is because modern engines are incredibly more sophisticated and more powerful than your old school 350s. Imagine trying to debug a variable valve timing solenoid, or injector clogging without a code that’s spit out to you. Near impossible. And I’m saying this as someone who drives a 1988 Ford bronco with a 351 Windsor swap. Don’t get me wrong. I love old school v8s but they are arguably worse in every way than even modern 4 bangers. … Sorry for the rant guys it’s a pet peeve of mine.

A lot of the iot infrastructure is meant to log once per minute or less, for 365 days/year. In my day job, I am an unmanned ground vehicle software engineer. I use ROS and we make a lot of data collections (we call them bags). It is surprisingly hard to keep a good record of what is going on. So we often collect a lot of extra information, like video and user button presses, to help us understand the context of what we have in a bag. I would say, at a minimum, you would want a video of the job, and a record of the job, possibly the original CAM project file.

The most interesting information (IMO) is when the motors skip steps, how much flex is in the gantry, and hot hot the chips are. Maybe the amount of power the spindle is using would be useful too.

You can (and we probably all should) keep a notebook and record the general settings, which bit you used, how long it took, and write some notes about how it went. But the automated system would have an advantage if you later noticed a phenomenon, and you wanted to see if it was present in previous jobs, you might have the data.

As for format, I would just drop it to a text file or csv. You can do a lot with a pi attached, reading from some sensors and writing it to a file. Then you can also attach photos or a video and have one clock.

Text files are by far most common in my field (cybersecurity). Just logs. Im going to speak with a few friends tomorrow to get some input from them.

Skipped steps do seem very important. I also think the power draw of the spindle and the rpm could be correlated to determine if/when a bit breaks. Which could signal an emergency stop.

The TMC drivers do know when they skip, and they do something nifty with the phase of the back emf or something to tell. This is how they do sensorless homing. They must know when they get close too, because you can adjust the limit for sensorless homing. I am not sure how you would get that data out of them, but it would be a cool thing to have.

I also had been surprised when found that they moved home assistant to hassos. But after i tried this docker based approach i realized that this way to implement plugin system is pretty nice. You will need esphome plugin to flash and configure your wifi sensor node devices.

Now about data collecting. It’s true that gasoline engines made a big step forward thanks to electronical controllers. But engine + controller is closed loop system that can be tuned with a lot of parameters.
CNC in most cases is open system. Motion controller send steps to steppers (or servo motors with local controllers) and doesn’t care about any feedback. Same for RPM control of a spindle. That’s because cnc based on predictable model and most people ok with that. Properly setup current on your steppers (even with smart trinamic drivers), respect to obvious limits (not to high feedrate) and all should be fine.

So as far as i study milling i realized that most important factor here is good quality of mechanics - cnc should be rigid and precise, tool should be sharp, etc. An second (yes, second - because without good mechanics it has no value) is how qualified is a human who prepare gcode in CAM and are operator of the CNC.

One reason I can see to monitor temperatures is if you want to use the maximum amount of current for your steppers without overheating them. Not really necessary but could be a fun project.

Measuring RPMs of the tool could be interesting in a PID controller (closed loop, not data science, and you can buy/build the kit already). Or, just monitor the RPMs and adjust your CAM based on what you learned from it.

TMC drivers can indeed detect missed steps, but not under all circumstances (for example at slow speeds). I am not aware of a feature in Marlin that can abort the job when a missed step is detected during milling.

I think that encoders on steppers would be a great safety feature. The forums are full of posts of people that experienced their Z axis dropping unexpectedly. I also had this problem. If you’re cutting wood, you’re likely to start a fire that way. Encoders would immediately measure that the stepper position is too far off and could be wired/programmed to activate the emergency stop immediately.

I agree that all should be fine with a properly set up machine and decent CAM, but it’s still a lot of fun to work on these kinds of ideas :slight_smile:

Good quality of mechanics is always going to improve performance but brute force is not the most efficient way to get there. Anyone can put big castings together for a stiff machine at a high price, but achieving a balance between price and performance requires skill.

The next level in that direction is software measurement and/or control, which shifts the cost into the software and instrumentation. Instrumentation (like encoders for example) has a cost. Software has a cost too but it is a different type of thing that scales differently.

I am pretty sure the way I use my machines I am leaving some performance on the table, not running as fast as I could, because I keep a conservative margin. With a strain gauge I could have a better understanding of what feed rates actually produce what deflections and pick a speed based on real data instead of trial and error, stopping the experiments when it’s “good enough” but still far from optimal.

I always think of the RC helicopters and planes my dad made when I was a kid. They were very expensive, and if they broke, you would have to replace most of them. The parts had to have high precision (expensive) to be able to move smoothly, but still fly. Now, a chunk of plastic with 4 motors is all you need, because we have control boards with integrated sensors, which you can buy for $25. The incremental software cost is free, because it is open source. You’ve traded a ton of mechanical complexity for some electrical complexity and a lot of software complexity. It scales really well.