3.3V Drop on Router Station Pro - instability

[lubosko]

Hi Ubinquiti,

I've bought Router Station Pro. It works fine, except stability of the system.
Configuration: Router Station Pro, Atheros AR7161 rev 2
- no modules in miniPCI slots
- 16 GB SD-Card
- 1GB USB flash
- power A: +16V (notebook adapter)
- power B: <+16, +24> Volt Laboratory power supply (transformer=not pulse) max. 5A
- oscilloscope 1GHz / 4 GSamples, scope probe 12pF/500MHz

I was wondering what's going on, therefore I look for power 3.3V and 5V on board. After while I've found an issue.
There are sporadic fall 3.3 Voltage out-off specification >2Volts.
What is more strange, that there are oscillations on 3.3 Volt cca on freq. 100MHz ~ 15-30 periods.
These drops on 3.3V appear sporadically cca <10-20> min. These drops appears no whole board, not only on some location of board.
For more details, please see pictures 1-3.
This finally explains instability whole system.

To invoke & observe this behavior on 3.3V I used 10Ohm resistor and sporadically increase power consumption on 3.3V. The behavior is absolutely same. As you can see on pictures, also oscillations are there ~100MHz with almost same no. of periods.
This proves that root cause of this behavior isn't processor or another component on board ...

Have you already solved this problem? Please send me proposal or solution.

Best Regards
Lubomir

[PJ]

Lubomir,
using 16VDC or 24VDC will not work
RouterStation Pro
Supports 39VDC to 56VDC for Power in
You can use either Passive POE or 802.3af 48V switch
to either Power JACK or Main RJ45 port

[lubosko]

Hi PJ,
as you recommended I powered RouterStation pro from Laboratory Power supply +50V, what is in range what you recommended.
I would be happy, if I can say, that problems with 3.3V - instability/oscillation are gone, but they remains.
Further investigations shows, that instability from 3.3V is affecting 5V too. Result is, that USB flash "key" (powered from 5V) sporadically disconnects too with all consequences such as data loss etc. .

I've made some consultations with other power supply experts, unfortunately no valuable sugestions without schematic.
Increasing capacity of "3.3V blocking" SMD capacitors C723, C725, C726, C727, C728, C1416, C1417, C1418, C1419, C1420, C147 by 100pF-10nF [COG/NPO Dielectric] doesn't help either. What was observed, that if board is powered in range >24V, then coild T7 is going through saturated state.

I'm really appreciate some sugestions/recomendations/patch to solve this problem.

Thank you in advance for sugestions and fruitful discussion.

Regards
Lubomir

[lubosko]

Hi PJ,

I followed your recommendation regarding replace C1421 FROM 1000pF to 400nF (0.47uF) 6.3V or higher 0402 size, any value form 100nf to 1uF should be sufficient. This will increase the current surge tolerance of the POE power supply.

I replaced C1421 [1nF] with [1uF X7R Dielectric] to see differencies in behavior.
The situation was improved only a little ...
See typical pictures (in attachment). As you can see from time-shoots on screenshots, 3.3V dropouts are sporadic, but not hard to catch.

As you can see, relatively small increasing power consumtion triggers oscillation.

1uF situation shows, that duration of oscillation takes longer time to stabilize in comparison with 1nF situation.

Further investigation shows, that oscillations are bigger (not longer) in case, when Wifi module from miniPCI slot is removed.
In case when only processor is powered (no miniPCI / no USB / no SD-Card), then oscillation is more significant and goes below 2.0V !

Another issue I found, when I play with RouterStation Pro during weekend :icon_wink:. If is applyed 2 streams of "ping flood" [ 1 stream = 200ms / 65000 length of data-packet], then temperature of processor was increased drastically, despite the processor load was cca 25% only.
[Platform: OpenWrt 2.6.30-10 with optimization -O3].
Therefore I exchanged processor cooler to regular one, not as is distributed by Ubiquiti. But this isn't really problem ... just info ...

Do you have some idea what's going on with this 3.3V drop-out?

Best Regards
Lubomir

[lubosko]

Dear Mr. Patrick Jabbaz & UBNT support team,

thank you, for measurements and screenshots what you did in your laboratory. I also checked soldering of U93 as you recommended. It is/was soldered properly.

About possibility to send you the RouterStation Pro board back to USA from Central Europe, it makes no sense. The delivery price is higher than price for an new RouterStation Pro board. But thank you for this offer anyway. There are no big business behind, I would like to use it at home ...

Now little bit about technical point:
As you can see from your screenshots and my screenshots, they are almost the same when scopes (your & my) are set with same parameters [1us time base]. Pic1-ubnt is your screenshots (from your laboratory), which shows, that 3.3V are within specification.
Pic2-19V shows situation, when board is powered by +19V. Pic3-50V shows situation, when board is powered by +50V.

As you can see all screenshots looks similar and definitely they are within +/- 10% of specification for 3.3V. I don't complain at all on that, but ... you can set trigger point to 3.0V or little below and wait cca 30 minutes and set scope time base at 50ns, then you can catch these drops what I'm talking about.

To speedup this process (not wait for processor increased power consumption) you could do following:

1.) make small cable cca 10 cm with 10 Ohm resistor
- we'll simulate increasing power consumption cca about 330 mA by this resistor on 3.3V power circuit.
2.) connect this cable to connector J3 [UART] pin GND
3.) connect scope's probe to connector J4 [JTAG] pins 3.3V & GND
4.) set scope
- time base [50 ns]
- volts [200 mV]
- trigger point [3.000 V]
- trigger mode [normal or single]
- trigger slope [falling edge]
5.) hold another ending of cable with 10 Ohm resistor in your hand and "touch" to J3 [UART] pin 3.3V. (increasing of the power consumption about cca 330 mA)
6.) "un-touch" pin 3.3V on connector J3 [UART]
7.) repeat points 5 - 6 few times and hopefully your scope will catch situation with drop 3.3V

I recommend, that you should to use scope with 4GSamples/s or more, because you will not be able to catch these drops on 3.3V power line.

Please make measurements when processor is running in normal mode, not in "Idle" state.

These drops would be fine for some slow speed processors, but drops cca [5 ns] on board with processor speed 720 MHz it's really problem. During these 5ns processor made cca 3 instructions!

What you observed now?
Have you catch these drops on 3.3V power line?

Best Regards
Lubomir

[PJ]

Lubomir,
For the detail power analysis the proper measurement should be right at the CPU under the IC back side of the board.
The measurements taken at the header pins, is not what the CPU sees.
The CPU has low impedance power plane, the JTAG and serial pin header most likely do not.

Any way I have not seen any problems related to power drops you described, it could be possible that You hardware has problems. Have You run any functional test? does it work, or are you getting Power cycles, and having reset issues?

[lubosko]

Hi PJ & support team,

yes you've generally right about proper measurement point(s), but the same peaks/drops I saw on all 3.3V "blocking" capacitors and also under CPU, therefore I choose header pins as easiest measure points.
Yesterday I loaded another firmware into board not "openwrt", but "dd-wrt". The situation is even worst. There aren't only some superposed 100MHz oscillations on generally stable 3.3V DC, but there appeared also declination of the 3.3V DC!
This declination of generally stable +3.3V DC in conjunction with "dd-wrt" firmware I've solved by tantalum capacitors 22-33uF connected in parallel with all "blocking" 3.3V capacitors on the board. Unfortunately even this wasn't enough...
I've to increase total "blocking capacity" about 3mF/6.3V to prevent declination of usually stable 3.3V DC.
Probably dd-wrt firmware is better for testing purpose ;).

About another questions what you asked:
Functional test? Do you've some?

No I don't have power cycles or reset issues.
The problems begin, when I recognized that:
- USB device(s) is disconnected from time to time, without any request or reason.
- corrupted data on USB-Flash disk
- On RS-232 terminal appears some strange characters from time to time. Then I've seen that some '1' goes to '0' or vice versa from time to time.
- flashing process of "firmware" seems, that finished OK, but data in flash was corrupted.

I'm afraid, that RMA isn't available for me anyway, because I put there another "blocking" capacitors, etc.
From this point of view, you probably will not accept it.

On the other hand I spend so much time to find root cause these strange 100MHz oscillations, unfortunatelly without success. Without schematics and layout is hard anyway.

It looks like great board, but...

Merry Christmas & Happy New Year from Central Europe.

Best Regards
Lubomir