I started writing this last night and wasn’t able to finish the response, but you’ve confirmed what I suspected. You are turning on color inks and using them as gray partitions without entering ink limits so they are using whatever your default ink limit is. I'm curious what you put in the density setting when doing that...
As Paul had indicated, with your ink set in the 3880 the only gray inks should be K, Lk, LLK. If you want to include the color inks you should set them as toner 1 and toner 2. A good place to start with the ink limits for the M/Lm and C/Lc inks are about 5-10. If you really want to cool it down you can set the limit as high as 15-20, but it is going to have a really strong color cast to it. I generally use the toners to give a slight selenium tone to the print, and that is generally achieved by using slightly more magenta than cyan. maybe 12-15 magenta and 7-10 cyan.
I think the biggest problem you're having is that the density setting is not what you think it is. The “Density” setting in the QTRgui is for determining where the lighter dilution stops and where the full strength ink starts. I call it the cross over point, but you might hear other people call it the insertion point, or relative gray value. That value is on a scale of 0-100. The darkest ink’s density or gray value is always 100, then each of the lighter dilutions are some percentage of that. These settings need to map very closely to the original density distribution of the unaltered K channel, but since there are only three gray inks in this ink set and since there is a lot of built in overlap you can be a little sloppy with this setting, and what i've included below is a good starting point the P800 and glossy papers. The same is true for the color inks in the toning settings. Since you are only using two inks, and since they are generally about a 1/3 dilution you can just use 30-35 as the “density" setting for the lighter of the two toner inks (assuming the ink limit is the same for the two dilutions for that color).
Here is what I have for ink limits and cross over points as a starting place for a couple different gloss papers (the the gray_Val= and toner_Val is the same as the Density setting in the QTRgui). This will make a more or less neutral feeling print.
######
LIMIT_K=55
BOOST_K=
LIMIT_C=4
LIMIT_M=5
LIMIT_Y=0
LIMIT_LC=4
LIMIT_LM=5
LIMIT_LK=55
LIMIT_LLK=55
# Gray Partitioning Information
N_OF_GRAY_PARTS=3
GRAY_INK_1=K
GRAY_VAL_1=100
GRAY_INK_2=LK
GRAY_VAL_2=35
GRAY_INK_3=LLK
GRAY_VAL_3=11
GRAY_HIGHLIGHT=8
GRAY_SHADOW=24
GRAY_GAMMA=1
GRAY_CURVE=
# Toner Partition Information
N_OF_TONER_PARTS=2
TONER_INK_1=C
TONER_VAL_1=100
TONER_INK_2=LC
TONER_VAL_2=36
TONER_HIGHLIGHT=3
TONER_SHADOW=4
TONER_GAMMA=1
TONER_CURVE=
N_OF_TONER_2_PARTS=2
TONER_2_INK_1=M
TONER_2_VAL_1=100
TONER_2_INK_2=LM
TONER_2_VAL_2=36
TONER_2_HIGHLIGHT=3
TONER_2_SHADOW=4
TONER_2_GAMMA=1
TONER_2_CURVE=
######
Hope that helps,
Richard Boutwell
http://www.richardboutwell.com/