myhdl-list

[myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

Hi Jan,

    I'm having difficulty using itertools objects in MyHDL simulations. 
It seems this is the case because, from some crazy reason, the Python 
itertools objects are *not* of type GeneratorType! In fact, it looks 
like each one is its own type! (eg itertools.izip, itertools.chain).

    This causes myhdl._Simulation._checkArgs to raise an "Inappropriate 
Argument Type" SimulationError.

    I'd really like to be able to use itertools.izip() and 
itertools.chain() at least. What could be done about the situation? Is 
there anyway to "coerce" the iterools into GeneratorTypes?

Thanks,
George

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

George Pantazopoulos wrote:
> Hi Jan,
>
>     I'm having difficulty using itertools objects in MyHDL simulations. 
> It seems this is the case because, from some crazy reason, the Python 
> itertools objects are *not* of type GeneratorType! In fact, it looks 
> like each one is its own type! (eg itertools.izip, itertools.chain).
>
>     This causes myhdl._Simulation._checkArgs to raise an "Inappropriate 
> Argument Type" SimulationError.
>
>     I'd really like to be able to use itertools.izip() and 
> itertools.chain() at least. What could be done about the situation? Is 
> there anyway to "coerce" the iterools into GeneratorTypes?
>
> Thanks,
>   
I found an interesting article on doing type checking in Python:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/305888

I'm not sure, but perhaps you could check for generator-like behavior 
(eg. a 'next' function) and not GeneratorType directly?

Thanks,
George

Re: [myhdl-list] difficulty using itertools with MyHDL

[Jan D. <ja...@ja...>]

George Pantazopoulos wrote:
> Hi Jan,
> 
>     I'm having difficulty using itertools objects in MyHDL simulations.
 >
> It seems this is the case because, from some crazy reason, the Python 
> itertools objects are *not* of type GeneratorType! In fact, it looks 
> like each one is its own type! (eg itertools.izip, itertools.chain).
> 
>     This causes myhdl._Simulation._checkArgs to raise an "Inappropriate 
> Argument Type" SimulationError.
> 
>     I'd really like to be able to use itertools.izip() and 
> itertools.chain() at least. What could be done about the situation? Is 
> there anyway to "coerce" the iterools into GeneratorTypes?

George:

You have hinted at this before, but until you post a concrete, simple
example, I think I'm not going to get it.

What are you trying to achieve? Coercing an iterable into a generator
would mean that you use it to describe MyHDL behavior. I just don't
see how that works.

If on the other hand, it's used to describe structure (like lists,
tuples or sets of generators), then you certainly don't want to
turn it into a generator. Look into _util._flatten - you want
to add the itertools types you want to use to the list of other
"structural" iterators.

But then still - why? I understand that itertools objects are
useful when dealing with a very large number of elements. I just
don't see how we can get at that point in structural MyHDL -
moreover, it only affects the elaboration phase, not the
simulation. What's the problem with using standard list
operations?

Jan

-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
     From Python to silicon:
     http://myhdl.jandecaluwe.com

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

I agree, I think I could have done a better job explaining what I wanted 
to do.

I want to run multi-stage simulations for automated unit testing.

For example, I have a ring buffer that I want to unit test.

For some tests I want to check the border case where its full and read 
or writes are done to it.
The first stage of the test involves filling it to capacity. The second 
stage involves driving signals and checking outputs.
There could be multiple tests that start off by filling it up, so in 
practice I'd want to reuse the filler function.

Stage 1:
   Fill ring buffer.

Stage 2:
   Simultaneously drive signals and check outputs


If I just return instances() then all the generators will be run 
together by the Simulation class, which is not what I want.
Thus, I use chain() to impose an ordering. For Stage 2 I use izip() to 
ensure that the driver and monitor are run in lockstep.

It goes something like this:

stage1 = filler()

stage2 = izip(driver, monitor)

complete_sequence = chain(stage1, stage2)


Right now I just copied and pasted the itertools functions from the 
Python docs into my code.
In this way they are generators, and not itertools objects. Perhaps this 
is because in the Python library they inherit from a superclass and I don't?

I've left out the definition of the Driver and Monitor, but they are 
ordinary
generators like the ones found in the test benches in your cookbooks.

    def bench():
      
        # I've deliberately glossed over the dut setup
        dut = RingBuffer(clk, rst, dat_i, we)
        clk = Signal(bool(0))
        clkgen = ClkGen(clk)

        def filler():
           
            # Fill up the Ring Buffer
            for i in range(depth):
               
                # Set up a write
                dat_i.next = i+10
                we.next = True
               
                yield clk.posedge
                yield clk.negedge
           
            # De-assert the write enable
            we.next = False
            yield clk.posedge
            yield clk.negedge
           
        # Itertools can't directly be used.
        # Seems that the _Simulation.py checks for GeneratorType, but 
itertools object are not of that
        # type. See one idea for a solution here:
        # http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/439359
       
        # duplicate of Python's built-in izip(), except this one is of 
type 'generator'
        def izip(*iterables):
            iterables = map(iter, iterables)
            while iterables:
                result = [it.next() for it in iterables]
                yield tuple(result)
       
        # Same goes for chain:
        def chain(*iterables):
            for it in iterables:
                for element in it:
                    yield element
       
      
        # Stage 1 - Fill the ring buffer
        stage1 = filler()
       
        # Stage 2 - Perform border-case testing
        # Simultaneously drive signals and monitor them 
        stage2 = izip(driver, monitor)
       
        # Make this look like a generator
        def stopsim():
            raise StopSimulation()
            yield None
           
        # Chain the generators so they run sequentially
        ordered_tests = chain(stage1, stage2, stopsim())
              
        return clkgen, dut, ordered_tests

# For py.test
def test_1():
    tb = traceSignals(bench())
    sim = Simulation(tb)
    sim.run()
 

Sorry for the confusion earlier. Does this help you understand what I 
want to accomplish? This is already working for me, except that I have 
to redefine izip() and ichain() myself. Perhaps you know of a better way 
to do this?

Thanks,
George

Re: [myhdl-list] difficulty using itertools with MyHDL

[Jan D. <ja...@ja...>]

George Pantazopoulos wrote:

> Sorry for the confusion earlier. Does this help you understand what I 
> want to accomplish? This is already working for me, except that I have 
> to redefine izip() and ichain() myself. Perhaps you know of a better way 
> to do this?

George:

Yes, I understand now. I think this qualifies as Truly Advanced Usage :-)
What happens is that you do want to use itertools to describe
behavior, not structure, in MyHDL terminology. You use the fact
that you can have a generator yield other generators, something
which I originally included to support bus-functional procedures.
With all this low-level RTL stuff that I'm doing these days, I
had all but forgotten about it :-)

I understand why you want to use itertools - not for performance,
but as a clear and concise way to manipulate iterators.

The missing part is how to make sure that the MyHDL simulator
handles your final iterator properly. There are 2 options:
either it should support that object directly, or it should be
converted to a generator first.

I have briefly looked at the first option - but this whole
typing system related to iterators seems complex. It's not
clear how to do it robustly. Moreover, it may not be wise:
perhaps someday somebody would want to use a similar object for
structure instead of behavior.

So at this point I would look for a workaround and convert
iterator objects to the "officially" supported types
(for behavior or structure) as desired. The advantage is
of course that no change to MyHDL would be required.

A workaround for your case could be the following. Just use
itertools to set up your iterator as desired. Then convert
it to a generator using a one-liner generator expression, e.g.:

ordered_tests = (t for t in chain(stage1, stage2, stopsim()))

This should work for any iterator you can imagine.

Regards,

Jan

-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
     From Python to silicon:
     http://myhdl.jandecaluwe.com

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

On Fri, October 20, 2006 10:51 am, Jan Decaluwe wrote:
> George Pantazopoulos wrote:
>
>> Sorry for the confusion earlier. Does this help you understand what I
>> want to accomplish? This is already working for me, except that I have
>> to redefine izip() and ichain() myself. Perhaps you know of a better w=
ay
>> to do this?
>
> George:
>
> Yes, I understand now. I think this qualifies as Truly Advanced Usage :=
-)

Thanks ;-) A real need arose as I was unit testing my ring buffer, and
this was my creative answer to it. Good stuff happens when you're having
fun (er, well not always) ;-)

>
> I understand why you want to use itertools - not for performance,
> but as a clear and concise way to manipulate iterators.
>
Exactly!

> So at this point I would look for a workaround and convert
> iterator objects to the "officially" supported types
> (for behavior or structure) as desired. The advantage is
> of course that no change to MyHDL would be required.
>
> A workaround for your case could be the following. Just use
> itertools to set up your iterator as desired. Then convert
> it to a generator using a one-liner generator expression, e.g.:
>
> ordered_tests =3D (t for t in chain(stage1, stage2, stopsim()))
>
> This should work for any iterator you can imagine.
>
That sonds like an excellent solution. Thanks!

George
http://www.gammaburst.net

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

>> So at this point I would look for a workaround and convert
>> iterator objects to the "officially" supported types
>> (for behavior or structure) as desired. The advantage is
>> of course that no change to MyHDL would be required.
>>
>> A workaround for your case could be the following. Just use
>> itertools to set up your iterator as desired. Then convert
>> it to a generator using a one-liner generator expression, e.g.:
>>
>> ordered_tests = (t for t in chain(stage1, stage2, stopsim()))
>>
>> This should work for any iterator you can imagine.
>>
>>     
Hi Jan, I like your solution, but I've hit a snag and I'm not sure why 
I'm getting the error message below. Could you perhaps enlighten me? ;-)

I've pasted a stripped-down example below:

Thanks,
George


$ python -i test__ascii_timing_spec.py
 >>> eggs()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "test__ascii_timing_spec.py", line 92, in eggs
    sim.run()
  File "/usr/lib/python2.4/site-packages/myhdl/_Simulation.py", line 
132, in run

    waiter.next(waiters, actives, exc)
  File "/usr/lib/python2.4/site-packages/myhdl/_Waiter.py", line 128, in 
next
    schedule((_simulator._time + clause._time, self))
AttributeError: 'tuple' object has no attribute '_time'
 >>>

# 
---------------------------------------------------------------------------

from myhdl import Signal, delay, Simulation, instance, instances

def eggs():
   
    def bench(sigs, cts):
       
        def driver(sigs):
           
            clk = sigs['clk']
           
            for i in range(10):
                yield delay(1)
                clk.next = not clk
       
        def monitor(sigs, cts):
           
            clk = sigs['clk']
           
            i = 0
            while True:
               
                yield delay(1)
                print "clk = " + str(clk)
           
                i += 1
                 
        import itertools
       
        # Execute the driver and monitor together
        combo = (t for t in itertools.izip(driver(sigs), monitor(sigs, 
cts)))
       
        return combo


    sigs = dict(clk=Signal(bool(0)))
    ref_cts = dict(clk = [1, 0, 1, 0, 1, 0, 1, 1,
                          0, 0, 1, 1, 0, 0, 1, 1,
                          1, 1, 0, 0, 0, 0, 1, 1,
                          1, 1, 0, 0, 0, 0, 1, 1,
                          1, 1, 0, 0, 0, 0, 1, 1])

    sim = Simulation(bench(sigs, ref_cts))
    sim.run()   

# 
---------------------------------------------------------------------------

Re: [myhdl-list] difficulty using itertools with MyHDL

[Jan D. <ja...@ja...>]

George Pantazopoulos wrote:
>>>So at this point I would look for a workaround and convert
>>>iterator objects to the "officially" supported types
>>>(for behavior or structure) as desired. The advantage is
>>>of course that no change to MyHDL would be required.
>>>
>>>A workaround for your case could be the following. Just use
>>>itertools to set up your iterator as desired. Then convert
>>>it to a generator using a one-liner generator expression, e.g.:
>>>
>>>ordered_tests = (t for t in chain(stage1, stage2, stopsim()))
>>>
>>>This should work for any iterator you can imagine.
>>>
>>>    
> 
> Hi Jan, I like your solution, but I've hit a snag and I'm not sure why 
> I'm getting the error message below. Could you perhaps enlighten me? ;-)

Ok - let's backtrack. You're still doing something different
from what I thought. Actually itertools introduce behavior which
I haven't anticipated (and is currently confusing me).

With a generator expression, you first "unroll" an iterator
and then re-pack it into a generator again. The iterator
in question is a MyHDL generator in your case - so after
"unrolling" it's done,  and that's not what you want.

The error message you get is because something like
   yield delay(1), delay(2)

is not supported by MyHDL (of course, it should be
flagged as an error instead of giving a traceback.) But the
real problem lies elsewhere (that is, the generators
are done already before starting the simulation.)

With itertools, the orignal generators stay "alive" but you
still manage to run their yield statements in "lockstep". This
is something I haven't anticipated. It "works" when using
a real generator (instead of a generator expression)
explicitly as follows:

         c = itertools.izip(driver(sigs), monitor(sigs, cts))
         @instance
         def combo():
             while True:
                 yield c.next()

         return combo

However, I'm really not sure that the interaction of this
with the MyHDL scheduler works meaningfully. Normally the
scheduler should decide when a MyHDL generator is run, and
this seems to inferere with that. I'll have to experiment
with itertools.

I'd also like to go back to what you actually want to
achieve. Running "in lockstep" in event-driven simulation
is best achieved by waiting on signal changes. This should
be possible using "conventional" simulator usage. Perhaps
there are other ways to achieve what you want, or at
least give more insight in the problem.

Jan

-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
     From Python to silicon:
     http://myhdl.jandecaluwe.com

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

Jan Decaluwe wrote:
> George Pantazopoulos wrote:
>   
>>>> So at this point I would look for a workaround and convert
>>>> iterator objects to the "officially" supported types
>>>> (for behavior or structure) as desired. The advantage is
>>>> of course that no change to MyHDL would be required.
>>>>
>>>> A workaround for your case could be the following. Just use
>>>> itertools to set up your iterator as desired. Then convert
>>>> it to a generator using a one-liner generator expression, e.g.:
>>>>
>>>> ordered_tests = (t for t in chain(stage1, stage2, stopsim()))
>>>>
>>>> This should work for any iterator you can imagine.
>>>>
>>>>    
>>>>         
>> Hi Jan, I like your solution, but I've hit a snag and I'm not sure why 
>> I'm getting the error message below. Could you perhaps enlighten me? ;-)
>>     
>
> Ok - let's backtrack. You're still doing something different
> from what I thought. Actually itertools introduce behavior which
> I haven't anticipated (and is currently confusing me).
>
> With a generator expression, you first "unroll" an iterator
> and then re-pack it into a generator again. The iterator
> in question is a MyHDL generator in your case - so after
> "unrolling" it's done,  and that's not what you want.
>
> The error message you get is because something like
>    yield delay(1), delay(2)
>
> is not supported by MyHDL (of course, it should be
> flagged as an error instead of giving a traceback.) But the
> real problem lies elsewhere (that is, the generators
> are done already before starting the simulation.)
>
> With itertools, the orignal generators stay "alive" but you
> still manage to run their yield statements in "lockstep". This
> is something I haven't anticipated. It "works" when using
> a real generator (instead of a generator expression)
> explicitly as follows:
>
>          c = itertools.izip(driver(sigs), monitor(sigs, cts))
>          @instance
>          def combo():
>              while True:
>                  yield c.next()
>
>          return combo
>
> However, I'm really not sure that the interaction of this
> with the MyHDL scheduler works meaningfully. Normally the
> scheduler should decide when a MyHDL generator is run, and
> this seems to inferere with that. I'll have to experiment
> with itertools.
>
> I'd also like to go back to what you actually want to
> achieve. Running "in lockstep" in event-driven simulation
> is best achieved by waiting on signal changes. This should
> be possible using "conventional" simulator usage. Perhaps
> there are other ways to achieve what you want, or at
> least give more insight in the problem.
>   
Hi Jan,
  
   I don't have time for a full reply at the moment, but in brief:

   1) I want to run a unit test where there are sequential "stages"

   2) Some of the stages need a driver/monitor pair running "in lockstep".
      The "driver" manipulates the signals going into the dut.
      The "monitor" checks the dut outputs.

      My coding efforts in this thread stem from the above points. Can I 
help you further by elaborating on any of my ideas? I'm confident you 
will come to an understanding soon.

Thanks,
George

> Jan
>
>   

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

>
>>> I'd also like to go back to what you actually want to
>>> achieve. Running "in lockstep" in event-driven simulation
>>> is best achieved by waiting on signal changes. This should
>>> be possible using "conventional" simulator usage. Perhaps
>>> there are other ways to achieve what you want, or at
>>> least give more insight in the problem.
>>>   
>>>       
> Hi Jan,
>   
>    I don't have time for a full reply at the moment, but in brief:
>
>    1) I want to run a unit test where there are sequential "stages"
>
>    2) Some of the stages need a driver/monitor pair running "in lockstep".
>       The "driver" manipulates the signals going into the dut.
>       The "monitor" checks the dut outputs.
>
>       My coding efforts in this thread stem from the above points. Can I 
> help you further by elaborating on any of my ideas? I'm confident you 
> will come to an understanding soon.
>
> Thanks,
> George
>
>   
Hi Jan,

   To elaborate, for a unit test I wish to parse a pair of ASCII Timing 
Spec (ATS) blocks into a driver/monitor combo. However before doing that 
I want to set up the device under test using a separate generator (in 
this example it fills up the ringbuffer to set up a border case). I'd 
like to be able to reuse this setup generator filler() across multiple 
unit tests.

I'm still tweaking the syntax and usage of the ASCII Timing Spec 
(feedback would be helpful here), but what I want looks something like this:

The ASCII Timing Spec specifies the signal states at each timestep, not 
just at edges. So I think my driver loop would involve doing something 
like yield delay(1) between setting each signal to its .next value. 
Likewise, the monitor would yield delay(1) between each iteration of 
checking the signals (according to the ATS):

def bench_1():

    driver_ats = \
    """
    Begin ASCII Timing Spec
   
    clk     = ----____----____----____----____
    rst     = --------________________________
       
    End ASCII Timing Spec
    """   

    monitor_ats = \
    """
    Begin ASCII Timing Spec

    count_o = X.......0.......1.......2.......

    End ASCII Timing Spec
    """


    sigs['clk'] = Signal(bool(0))
    sigs['rst'] = Signal(bool(0))
    sigs['count_o'] = Signal(intbv(0)[4:])

    dut = ringbuffer(clk     = sigs['clk'],
                     rst     = sigs['rst'],
                     count_o = sigs['count_o'])

    # Manipuate the dut in two stages.
    # Stage 1: performs setup (eg fills the ringbuffer)

    # Stage 2: drive inputs and check outputs.
    #          Tests a boundary case that involves a full ringbuffer.

    stage1  = filler()

    driver  = drive_signals_using_ats(driver_ats,  sigs)
    monitor = check_signals_using_ats(monitor_ats, sigs)

    stage2  = izip(driver(), monitor())

    combo = chain(stage1, stage2, stopsim())

    return combo

# ---------------------------

def test():

    sim = Simulation(bench_1())
    sim.run()


Let me know what else you might need to understand what I want to do. 
The ATS is intended to make unit tests more fun to write and maintain.

Thanks,
George

   

Re: [myhdl-list] difficulty using itertools with MyHDL

[George P. <ge...@ga...>]

> 
> def bench_1():
> 
>     driver_ats = \
>     """
>     Begin ASCII Timing Spec
>    
>     clk     = ----____----____----____----____
>     rst     = --------________________________
>        
>     End ASCII Timing Spec
>     """   
> 
>     monitor_ats = \
>     """
>     Begin ASCII Timing Spec
> 
>     count_o = X.......0.......1.......2.......
> 
>     End ASCII Timing Spec
>     """
> 
> 
>     sigs['clk'] = Signal(bool(0))
>     sigs['rst'] = Signal(bool(0))
>     sigs['count_o'] = Signal(intbv(0)[4:])
> 
>     dut = ringbuffer(clk     = sigs['clk'],
>                      rst     = sigs['rst'],
>                      count_o = sigs['count_o'])
> 
>     # Manipuate the dut in two stages.
>     # Stage 1: performs setup (eg fills the ringbuffer)
> 
>     # Stage 2: drive inputs and check outputs.
>     #          Tests a boundary case that involves a full ringbuffer.
> 
>     stage1  = filler()
> 
>     driver  = drive_signals_using_ats(driver_ats,  sigs)
>     monitor = check_signals_using_ats(monitor_ats, sigs)
> 
>     stage2  = izip(driver(), monitor())
> 
>     combo = chain(stage1, stage2, stopsim())
> 
>     return combo
> 
> # ---------------------------
> 
> def test():
> 
>     sim = Simulation(bench_1())
>     sim.run()
> 
> 
Just a quick note.. I accidentally mixed a up-counter example and a ringbuffer example here, but the principle is the same

George

Re: [myhdl-list] difficulty using itertools with MyHDL

[Jan D. <ja...@ja...>]

Jan Decaluwe wrote:

> However, I'm really not sure that the interaction of this
> with the MyHDL scheduler works meaningfully. Normally the
> scheduler should decide when a MyHDL generator is run, and
> this seems to inferere with that. I'll have to experiment
> with itertools.

Ok, I'll start with the conclusion. itertools shouldn't be
used to manipulate MyHDL generators. The results are meaningless.

The reason is the following. When using the MyHDL simulator, there
is an implicit contract about (MyHDL) generator usage: don't
run them yourself (by calling their 'next') method, but let the
Simulator do that. Of course this is Python and I don't see how
this could be enforced. But the results are basically meaningless.

When using itertools in the way discussed, that's exactly what
happens under the hood - itertool object methods such as 'next'
run the underlying generators themselves.

No need for despair though. The MyHDL scheduler, combined with
coding techniques, should be able to support the required behavior
correctly, and probably in an easier way.

Here's an example. Consider this generator:

def gen(msg, d=1, n=3):
     for i in range(n):
         yield delay(d)
         print "  %s: %s" % (now(), msg)
     yield delay(1)

It prints out a message a number of times after a delay. Now
let's a play a game of ping-pong using itertools. We set
up the sequence as follows:

seq = itertools.izip(gen("ping", 1), gen("pong", 1))

and simulate this generator:

def test(seq):
     while seq:
         yield seq.next()

The result is:

   1: ping
   1: pong
   2: ping
   2: pong
   3: ping
   3: pong

Seems OK. But now let's try this:

seq = itertools.izip(gen("ping", 1), gen("pong", 4))

We give a much larger delay to "pong", and we would expect a
number of pings first. But we get:

   1: ping
   1: pong
   2: ping
   2: pong
   3: ping
   3: pong

No difference. Clearly not OK. The reason is that both generators
are incorrectly forced to run both as soon as the fastest one
triggers. Not good.

What to do? Well, one thing we can do is use the MyHDL simulator's
ability to schedule generators dynamically. If we simply simulate:

def test2():
     yield gen("ping", 1), gen("pong", 1)

we get:

   1: pong
   1: ping
   2: pong
   2: ping
   3: pong
   3: ping

as before. But when we now do:

def test2():
     yield gen("ping", 1), gen("pong", 4)

we get:

   1: ping
   2: ping
   3: ping
   4: pong
   8: pong
   12: pong

as expected.

There's more to it, but this is probably enough food for thought for
one post. More info can be found in the manual under
"bus-functional procedures".

Jan

-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
     From Python to silicon:
     http://myhdl.jandecaluwe.com