Isco D-Series Pump Controller firmware bugs

[banksey255]

Teledyne Isco are not able to provide support for their D-Series pump controllers due to no longer having any employees with the required knowledge. Therefore, I am hoping that someone on this board has encountered and devised a solution to a firmware bug that I recently discovered. To help others who may encounter this equipment, I will also document a second firmware bug that I am aware of and the solution I have devised for it at the end of this post. To be clear, I remotely control these devices using their serial interface.

The firmware bug that I have discovered that I want a solution for to prevent it occurring is that Isco D-Series pump controllers will (seemingly) randomly not move the pump connected to connector A, despite being commanded a non-zero refill rate or independent constant flow rate and the G& command reporting it is not stopped (I have observed this problem with multiple pumps and controllers running revision M firmware). This is contradicted by the STATUS command that reports it is stopped. If the controller is again commanded to begin motion, it reports an error that it is already running. The only way I have found to recover is to send the STOP command. Currently I have formulated a detection approach based on the aforementioned contradiction between the G& and STATUS command responses, however what I want is a solution that prevents the problem from occurring in the first place. My efforts to that end have been unsuccessful as I have been unable to identify any patterns for what causes it to occur (I have experimented with different command sequences and timings, but so far identified nothing). My code that sends commands to the pump does so to control a process variable using a PID algorithm and frequently switches the controller between independent constant flow mode and refill mode based on the sign of the PID algorithm output and sets the corresponding flow or refill rate to the magnitude of the PID algorithm output, with the problem occurring very roughly at a rate of once per 24 hours. There is the possibility that the problem occurs in other modes of operation, which I haven't investigated yet.

The second firmware bug that I will document here to help others is that if a pump is stopped then the refill rate is set less than 0.00011 mL/min and then the pump is commanded to begin refilling, it will begin refilling at maximum speed then shortly thereafter report a position error and stop. In contrast, if the pump is refilling, then the refill rate can be set less than 0.00011 mL/min without problem. Therefore, the solution to prevent this problem occurring is to simply never start refilling with the refill rate set less than 0.00011 mL/min, and then adjust the refill rate to whatever value is desired once refilling has commenced.

[banksey255]

I suppose I may be out of luck.

We continued to investigate the problem, which had been occurring on their firmware revision M, and we subsequently found also occurred on their "demo 79" revision firmware, which apparently was the last firmware released for that generation controller that they discontinued in 2013. We then tried an Isco D-series 4th generation controller running version 1.10 firmware and found it also had the same problem, and whilst they discontinued the 4th generation model in 2022, our hope is that it is recent enough that they can identify the cause and a solution.

If it's helpful to anyone, we also discovered a new firmware problem on their 4th generation controller where, intermittently, when pump B is running and switched from INDEPENDENT CONST FLOW to REFILL mode, or vice versa, it will continue to maintain the flow from the original mode, despite all commands reporting that the pump is operating in the new mode. The only way we've found to recover is to send the stop command.

[banksey255]

Here is a final summary of the Isco D-series pump controller problems we discovered when using the serial interface. If anyone ever discovers additional problems, please post to this thread as I want to know. Also, if anyone ever finds a solution to either problems (2) and/or (3) that prevents them occurring in the first place, in contrast to my solutions that detect and recover from them once they occur, I would very much like to know also. The problems are:

1. A pump will begin refilling at the maximum rate and shortly thereafter stop reporting a position error if started with a refill rate less than 0.00011 mL/min (110 nL/min). This problem has been observed in 4^th generation and older controllers. My solution is to simply never start refilling with a rate less than this, and then once refilling, set to whatever rate is desired.
2. Intermittently, pump A will not flow in REFILL or INDEPENDENT CONST FLOW mode. Other symptoms are a contradiction between STATUS reporting stopped despite G& reporting not stopped and the pump rejecting commands to run reporting that it is already running. The only way to recover is to stop the pump. This problem has been observed in 4^th generation and older controllers. My solution is if the flow is zero and the volume doesn't change for two successive G& commands, then simply stop then start the pump (although I suspect this may result in some false positives, so a more reliable test may be to instead see if there is a STATUS command contradiction, which I don't do simply as the time to execute the STATUS command is comparable to the combined time for commanding the pump to stop then start).
3. Intermittently, pumps B, C, and D do not switch from REFILL to INDEPENDENT CONST FLOW mode, or vice versa, and instead maintain flow in the original mode, despite the G&, MODE, and STATUS commands all reporting operation in the new mode. The only way to recover is to stop the pump. This problem has been observed in 4^th generation controllers. My solution is to compare the sign of the reported flow rate to that commanded, and if there is a disagreement, then simply stop then start the pump.

I suspect both problems (2) and (3) are associated with switching between REFILL and INDEPENDENT CONST FLOW modes. The very limited support I could obtain from Isco was a suspicion that it was associated with my code switching the modes as fast the Isco controller would allow and that a solution may be to increase the interval between commands, however my limited attempts to prove this failed (I inserted a delay between stopping one mode and starting the next that automatically increased by 25ms each time a problem was detected, but problems still occurred even once this delay had reach 150ms, which seemed already excessive given that it takes 25ms for it to respond to commands at its fastest baud rate, so I ended the investigation). If anyone wants to try to replicate either of these two problems, a method I used was to simply sinusoidally oscillate the flow rate with a one minute period and small 0.025 mL/min amplitude, and then simply log the data and look for a half period where the flow rate doesn't respond (in my experience this occurred within three hours of starting).

For reference, the older generation (three pump) controllers I ran tests with were running either revision M or Demo 79 firmware (which Isco informed me was the latest available for that generation), and the 4th generation (four pump) controllers were running version 1.10 firmware.

[banksey255]

We discovered more Isco D-series controller problems. Continuing the numbering from my previous post:

   4. Contrary to the manual that states up to seven decimal places are accepted for flow and refill rate setpoint commands, the device truncates to five decimal places, so less than 1e-5 mL/min is treated as zero. For me, this resulted in a scenario where my detection of problem (3) in my previous post failed because the flow rate was zero. My solution is to set minimum independent flow and refill rates of 1e-5 mL/min.

   5. Intermittently, a pump will stop for no reason. This has been observed in INDEPENDENT CONST FLOW, INDEPENDENT CONST PRESS, and REFILL modes. My solution is to monitor the operation status and if it reports a pump is stopped when it isn't supposed be, then command it to start.

   6. If a motor failure occurs, then the Isco controller disables the motor drives of all pumps by turning off a transistor and reports the status of all pumps as motor failure. This is expected behaviour, however whilst the documentation states that to recover the front panel switch must be set to standby and then to on to reset the system, we discovered an undocumented RESET command that accomplishes the same, thereby allowing recovery through the serial interface.

   7. If operating in INDEPENDENT CONST PRESS mode, and the pressure setpoint is changed by a small amount, we have observed that the Isco controller adjusts its pressure reading rather than adjusting its flow rate to target the new setpoint. We haven't explored this thoroughly, and there is a suspicion that it may be related to the PRESS. DEADBAND setting being turned on.

   8. If a pressure alarm occurs, then the device does not respond to flow commands as expected. The documentation doesn't say how to recover from this (adjusting pressure limits doesn't work) and the only way we have identified is by using the stop button on the device's front panel (that is, there appears to be no way to clear it through the serial interface). Our solution is to consider setting the pressure limits to values that are unlikely to be crossed and using our application communicating with the Isco controller to stop the pump should the pressure go outside an acceptable range instead of having the Isco controller automatically do this.

Beyond this, we have also encountered problems that are limitations rather than a result of not operating as intended:

   9. To control pressure, the Isco controllers use proportional-integral control, but the control gains cannot be adjusted, only the integral action can be turned on or off. As a result, we have experienced unstable or highly underdamped responses when the integral action is turned on, and unacceptably slow responses when the integral control is turned off.

Further to this, we have recently begun attempting to run Isco controllers in continuous operation modes where two pumps are paired and have experienced more difficulties, such as the Isco controller not starting pumps despite not reporting any problems.

As per previous posts, I continue to share the problems we discover and our solutions in case it ever helps anyone else as Teledyne Isco is unable to provide any support.

[banksey255]

Firstly, I want to mention that all the problems I have discovered also likely exist in Teledyne Isco's current pump controller they sell (the SyriXus) as I have been informed that those pumps use the same firmware as the older D-series controllers with an inconsequential change, yet despite currently selling that model, Teledyne Isco informed me that they are also unable to provide support for even it.

Ok, so I have a new problem and solution to report that we encountered due to undocumented behaviour resulting in guesswork for how to configure the Isco controller to operate as desired:

   10. If trying to run in CONTIN CONST FLOW mode then be aware that if set to DELIVER, then the flow rate setpoint must be set using the FLOW= command, whereas if set it to RECEIVE, then the flow rate setpoint must be set using the REFILL= command (or, if the dual pumps are C and D, then use FLOWC= and REFILLC=, respectively). The RUN command is used regardless (that is, in this case, don't use the REFILL command to start the pump when the REFILLL= command has been used). I have verified that the refill rate refill bug does not exist, which is problem (1) in a previous post.