s390 hardware config

[Bastian Blank]

Hi folks

I finaly finished a rudimentary hardware configuration. It have some
similarities with the redhat and suse sysconfig. (This was the simplest
schema I found.)

You can find the sources via
https://lophos.multibuild.org/svn/sysconfig/trunk.

It currently lacks documentation, but the important things are:
- it only works with udev for automatic device setup.
- configs are located in /etc/sysconfig/hardware, the names match
config-$BUS-$BUSID (for example config-ccw-0.0.9000).
ccwgroup also uses ccw for the config name.
- A config for ccwgroup contains a CCWGROUP_CHANS array:
/etc/sysconfig/hardware/config-ccw-0.0.9000:
| CCWGROUP_CHANS=(0.0.9000 0.0.9001 0.0.9002)
- A config for zfcp contains a ZFCP_DEVICES array:
/etc/sysconfig/hardware/config-ccw-0.0.2900:
| ZFCP_DEVICES=(0xXXXXXXXXXXXXXXXX:0x0000000000000000
- dasd is not yet supported.

Bastian

[Frans Pop]

Post by Bastian Blank
I finaly finished a rudimentary hardware configuration. It have some
similarities with the redhat and suse sysconfig. (This was the simplest
schema I found.)
| CCWGROUP_CHANS=(0.0.9000 0.0.9001 0.0.9002)

I now have 2.6.16 booting nicely on hercules and can get the net up by:
echo "0.0.0a00,0.0.0a01" >/sys/bus/ccwgroup/drivers/ctc/group
echo "1" >/sys/bus/ccwgroup/devices/0.0.0a00/online

I've not been able get udev to set up the hardware though.
I've created a file '/etc/sysconfig/hardware/config-ccw-0.0.0a00' with:
CCWGROUP_CHANS=(0.0.0a00 0.0.0a01)
but that does not seem to do anything.

By googling I found references to CCW_CHAN_IDS, so I also tried:
CCW_CHAN_IDS=(0.0.0a00 0.0.0a01)
but that did not work either.

Help or tips for debugging welcome.

[Bastian Blank]

Post by Frans Pop
I've not been able get udev to set up the hardware though.
CCWGROUP_CHANS=(0.0.0a00 0.0.0a01)
but that does not seem to do anything.

Hmm, I think I know the problem. I never tested CTC properly as it is
deprecated.

The kernel defines that the ctc channels are managed by cu3088 modules, which
is correct from the technical point of view. A CTC device on top needs
the additional ctc module. I think I should just add an explicit loading
of the module.

After loading the ctc module, a hwup ccw 0.0.0a00 works fine.

Bastian

[Frans Pop]

Post by Bastian Blank
After loading the ctc module, a hwup ccw 0.0.0a00 works fine.

The ctc module is loaded. I have it in /etc/modules.
A manual 'hwup ccw 0.0.0a00' does indeed do the trick.

The problem seems to be that loading the ctc module does not trigger the
proper events.

Post by Bastian Blank
I think I should just add an explicit loading of the module.

I guess you mean that ctc will be loaded as part of the processing of
/sys/bus/ccw/devices/0.0.0a00 so loading it "manually" in /etc/modules is
no longer needed?

Thanks.

[Bastian Blank]

Post by Frans Pop
I guess you mean that ctc will be loaded as part of the processing of
/sys/bus/ccw/devices/0.0.0a00 so loading it "manually" in /etc/modules is
no longer needed?

Exactly.

But there is another problem left: A race condition happens if only
0.0.0a00 is available but 0.0.0a01 not. If the later appears, it can't
find the proper config to use.

Bastian

[Adam Thornton]

Post by Bastian Blank

Post by Frans Pop
I guess you mean that ctc will be loaded as part of the processing of
/sys/bus/ccw/devices/0.0.0a00 so loading it "manually" in /etc/
modules is
no longer needed?

Exactly.
But there is another problem left: A race condition happens if only
0.0.0a00 is available but 0.0.0a01 not. If the later appears, it can't
find the proper config to use.

Have we seen this in practice?

Because of the way the CTC driver is structured, one address is input
and one is output if you're speaking IP over it. It's not like using
CTC for RSCS where each address is bidirectional. So if only one of
the pair is there, I'd say that it's broken and you shouldn't try to
use it as a network device. Now if it comes up later, it would be
nice to autodetect that and tell that you now had a CTC pair and try
to start the interface...but that might be a lot more trouble than
it's worth. If your CTCs are flapping, something's clearly pretty
wrong already.

Adam

[Ivan Warren]

Post by Adam Thornton
Have we seen this in practice?
Because of the way the CTC driver is structured, one address is input
and one is output if you're speaking IP over it. It's not like using
CTC for RSCS where each address is bidirectional. So if only one of the
pair is there, I'd say that it's broken and you shouldn't try to use it
as a network device. Now if it comes up later, it would be nice to
autodetect that and tell that you now had a CTC pair and try to start
the interface...but that might be a lot more trouble than it's worth.
If your CTCs are flapping, something's clearly pretty wrong already.
Adam

Not 100% sure Adam.. Bastian may have a point here..

Assuming the CTC pair (comprising the link to a TCP/IP virtual machine
or as a hercules means to talk outside) could be defined dynamically at
run time, I see where a problem may exist.

1st, when running under VM, CTCs have to be defined one after the other
(CP DEFINE CTC 0A00#CP DEFINE CTC 0A01 - followed by the appropriate
'couple' commands)

2nd under hercules, you either attach both CTCs as a group such as
"attach 0A00.2 CTCI ..." or independently (attach 0A00 CTCI .. attach
0A01 CTCI ...)

In any case, the CRWs are going to be made pending and presented one
after the other.. Any hotplugging technology will then receive a report
for the 1st instance followed by a report for the 2nd instance.

So it's clearly a possibility for a time window to exist for only the
first of the 2 exposures to either 'exist' or be known to the operating
system aven though the second exposure will eventually get there.

I doubt using some time lapse will do either.. This definitely doesn't
cover the fact of manually defining each CTC independently and manually.

This is only an issue for devices that start to exist *AFTER* IPL
(hotplugged devices).

(just my $.02..)

--Ivan

[Stephen Frazier]

It would seem that for a "hot plug" CTC device. You would set it up to
test for the presence of the other one of the pair. If they are not both
present then dont do anything. Thus in your example when 0A00 became
available nothing would hapen as 0A01 was not available. Then when 0A01
became available you could then activate 0A00.

Post by Ivan Warren

Post by Adam Thornton
Have we seen this in practice?
Because of the way the CTC driver is structured, one address is input
and one is output if you're speaking IP over it. It's not like using
CTC for RSCS where each address is bidirectional. So if only one of
the pair is there, I'd say that it's broken and you shouldn't try to
use it as a network device. Now if it comes up later, it would be
nice to autodetect that and tell that you now had a CTC pair and try
to start the interface...but that might be a lot more trouble than
it's worth. If your CTCs are flapping, something's clearly pretty
wrong already.
Adam

Not 100% sure Adam.. Bastian may have a point here..
Assuming the CTC pair (comprising the link to a TCP/IP virtual machine
or as a hercules means to talk outside) could be defined dynamically at
run time, I see where a problem may exist.
1st, when running under VM, CTCs have to be defined one after the other
(CP DEFINE CTC 0A00#CP DEFINE CTC 0A01 - followed by the appropriate
'couple' commands)
2nd under hercules, you either attach both CTCs as a group such as
"attach 0A00.2 CTCI ..." or independently (attach 0A00 CTCI .. attach
0A01 CTCI ...)
In any case, the CRWs are going to be made pending and presented one
after the other.. Any hotplugging technology will then receive a report
for the 1st instance followed by a report for the 2nd instance.
So it's clearly a possibility for a time window to exist for only the
first of the 2 exposures to either 'exist' or be known to the operating
system aven though the second exposure will eventually get there.
I doubt using some time lapse will do either.. This definitely doesn't
cover the fact of manually defining each CTC independently and manually.
This is only an issue for devices that start to exist *AFTER* IPL
(hotplugged devices).
(just my $.02..)
--Ivan

--
Stephen Frazier
Information Technology Unit
Oklahoma Department of Corrections
3400 Martin Luther King
Oklahoma City, Ok, 73111-4298
Tel.: [phone]
Fax: [phone]
Pager: [phone]
email: stevef%doc.state.ok.us

[Adam Thornton]

Post by Stephen Frazier
It would seem that for a "hot plug" CTC device. You would set it up
to test for the presence of the other one of the pair. If they are
not both present then dont do anything. Thus in your example when
0A00 became available nothing would hapen as 0A01 was not
available. Then when 0A01 became available you could then activate
0A00.

I'm going to be devil's advocate here:

*I* always use the even address as the bottom one of the pair.

Is there anything mandating that that has to be the case?

Adam

[Ivan Warren]

Post by Adam Thornton
*I* always use the even address as the bottom one of the pair.
Is there anything mandating that that has to be the case?
Adam

Oh.. You're lacking imagination ! You could push that one just a tad
further !

1) Use non consecutive pairs (Read=0a00, Write=0a10)
2) Reverse the order (Read=0a01, Write=0a00)
3) Set the pair on different channels (Read=0a00, Write=0b00)
4) Set the pair on different LCSS (Read=0:0a00, Write=1:0a00)
5) What you said : start with even (Read=0a01, Write=0a02)
6) Really wicked ! All of the above ! (Read=1:0b11, Write = 0:0a00)

But eventually, to make it work in a hot plug environment, you're going
to have to set some rules if you want the hotplug system to figure out
the relationship on its own (A: I'm doing something weird and it doesn't
work.. B: Don't do something weird) !

--Ivan

[Adam Thornton]

Post by Ivan Warren
Oh.. You're lacking imagination ! You could push that one just a
tad further !

I blame the total lack of cough syrup tonight for my failure of
imagination.

Adam

[Stephen Frazier]

When using a real CTC, that is a wire cable between two separate
computers, (I haven't seen one of those in years either but they do
really exist) it is necessary to reverse the order on one. One machine
read=0A00 write=0A01 the other read=0A01 write=0A00. One machine has to
read what the other is writing. I learned this in 1968 when I plugged
the tape drive cable from one CDC G-15 into another CDC G-15 instead of
a tape drive. If I issued a write tape from one machine and a read tape
on the other it would copy a buffer of data from one machines memory
into the other.

Post by Ivan Warren

Post by Adam Thornton
*I* always use the even address as the bottom one of the pair.
Is there anything mandating that that has to be the case?
Adam

Oh.. You're lacking imagination ! You could push that one just a tad
further !
1) Use non consecutive pairs (Read=0a00, Write=0a10)
2) Reverse the order (Read=0a01, Write=0a00)
3) Set the pair on different channels (Read=0a00, Write=0b00)
4) Set the pair on different LCSS (Read=0:0a00, Write=1:0a00)
5) What you said : start with even (Read=0a01, Write=0a02)
6) Really wicked ! All of the above ! (Read=1:0b11, Write = 0:0a00)
But eventually, to make it work in a hot plug environment, you're going
to have to set some rules if you want the hotplug system to figure out
the relationship on its own (A: I'm doing something weird and it doesn't
work.. B: Don't do something weird) !
--Ivan

--
Stephen Frazier
Information Technology Unit
Oklahoma Department of Corrections
3400 Martin Luther King
Oklahoma City, Ok, 73111-4298
Tel.: [phone]
Fax: [phone]
Pager: [phone]
email: stevef%doc.state.ok.us

[Bastian Blank]

Post by Stephen Frazier
It would seem that for a "hot plug" CTC device. You would set it up to
test for the presence of the other one of the pair. If they are not both
present then dont do anything. Thus in your example when 0A00 became
available nothing would hapen as 0A01 was not available. Then when 0A01
became available you could then activate 0A00.

You don't have the informations currently to see 0a01 as part of this
ctc device.

Bastian

[Bastian Blank]

Post by Frans Pop
echo "0.0.0a00,0.0.0a01" >/sys/bus/ccwgroup/drivers/ctc/group
echo "1" >/sys/bus/ccwgroup/devices/0.0.0a00/online

Can you please run:
# udevcontrol log_priority=debug
# echo /sys/bus/ccw/devices/0.0.0a00/uevent

and grep for hwup in syslog?

Maybe you should also upgrade to sysconfig-hardware 0.2, which is not
yet released.

Bastian

[Bastian Blank]

Post by Bastian Blank
# echo /sys/bus/ccw/devices/0.0.0a00/uevent

Should be

Post by Bastian Blank
# echo > /sys/bus/ccw/devices/0.0.0a00/uevent

Bastian

[Frans Pop]

Sorry for the delay.

Post by Bastian Blank
# udevcontrol log_priority=debug
# echo >/sys/bus/ccw/devices/0.0.0a00/uevent

If I run it after a clean boot (with my current workaround to bring up
ctc0 [1] disabled), I get:
udevd-event[2009]: run_program: '/sbin/hwup -A net ctc0 /class/net/ctc0'
udevd-event[2009]: run_program: '/sbin/hwup' returned with status 9

If I run it when the interface is already up, I get:
udevd-event[2031]: run_program: '/sbin/hwup -A ccw 0.0.0a00 /devices/css0/0.0.0007/0.0.0a00'
udevd-event[2031]: run_program: '/sbin/hwup' returned with status 1


[1] My current workaround is a local init.d script that contains:
#! /bin/sh
if uname -a | grep -q "2\.6"; then
hwup ccw 0.0.0a00
fi