Re: [myhdl-list] MEP - keep hierarchy in conversion

[David H. <da...@ad...>]

tl;dr: My patch against 0.8 appears can be applied against the 0.9-dev
repo, and it satisfies my very limited testing.

The changes in 0.9-dev were not near my hack-n-slash campaign.  (I kept
most of my changes _after_ MyHDL's normal extraction and analysis
algorithm, with only a few small hooks into signals and generators to
assist with partition-tagging.)

To test, I first removed the current myhdl-0.8 I had in my Python
installation, then I followed these steps:

# (I removed my current myhdl-0.8 from my Python installation.)
hg clone https://bitbucket.org/jandecaluwe/myhdl
cd myhdl
gunzip -c ../myhdl-0.8-partitions.patch.gz | patch -p1
python setup.py build
python setup.py install --user -O2

Then I reran my test_partition.py, and the output is identical.  (except
that the new myhdl is injecting some $signed(...) stuff in 1 place where
I'm substracting...)

I also reran my pcie designs (with partitions), and that also produced
identical output.  (but again with $signed(...) stuff but in 2 places where
I'm subtracting...)


To confirm I'm using the latest repo, here's the recent log entry:
hg log | head -8
changeset:   1358:324a8d3206fb
branch:      0.9-dev
tag:         tip
parent:      1356:db95c2d47119
parent:      1357:8de9f8771395
user:        Jan Decaluwe <ja...@ja...>
date:        Sun Sep 22 19:14:21 2013 +0200
summary:     Bug fix merge from default


And the patch applied fairly cleanly, with only 2 offset changes:
gunzip -c ../myhdl-0.8-partitions.patch.gz | patch -p1
patching file myhdl/_Signal.py
patching file myhdl/__init__.py
patching file myhdl/_always.py
patching file myhdl/_always_comb.py
Hunk #1 succeeded at 190 (offset 10 lines).
patching file myhdl/_always_seq.py
patching file myhdl/_contextdecorator.py
patching file myhdl/_extractHierarchy.py
patching file myhdl/_instance.py
patching file myhdl/_partition.py
patching file myhdl/conversion/_analyze.py
Hunk #5 succeeded at 865 (offset 5 lines).
patching file myhdl/conversion/_toVerilog.py


The rest of this email is a quick summary of my changes per file:

__init__.py: include "partition" and "get_current_partition" in the myhdl
namespace.  get_current_partition() may be called from within a partition
to assign attributes.

For example, to assign attributes using the "with" syntax:

with partition('A') as this_part:  # This will be renamed to A0
 this_part.attr_add('KEEP_HIERARCHY', 'TRUE')
 this_part.attr_add('DONT_TOUCH', 'TRUE')

@always(clk.posedge)
def clgd():
 d.next = c + e

And to assign attributes from inside a partition (such as when using the
"decorator" syntax):

@partition('AES') # <---- partition a la decorator.
def my_module_bob(...):
 # ...
 this_part = get_current_partition()
 this_part.attr_add('DONT_TOUCH', 'TRUE')
 return instances()


_Signal.py: Small change.  I radio tag each signal with the current
partition.  (Actually, I originally didn't need to do this, until I tried
figuring out where to place ConcatSignal drivers (ie, which partition to
invoke their _toVerilog [or _toVHDL] method).  In short, Iw as tired after
devising the auto-magic placement for every other signal type, so I just
gave up on auto-placement of ConcatSignals, and instead just use the tagged
partition to indicate where to place each ConcatSignal's driver.  (ie, For
all other signals, it doesn't matter where you happen to declare them,
since the output code will auto placed based on the location of any driving
generator.  But for ConcatSignal's, the output code will strictly place
their driver inside whatever partition you happend to create the
ConcatSignal in python.)


_instance.py: Small change, to support any extension to "monkey patch" its
way into tagging generators inheriting from _Instantiator.  (Actually, I
should've structured my change to _Signal.py like this one, to open the way
for any future extensions to also "monkey patch" their way into _Signal's
and shadow signals...)


_always.py:  _Always inherits from _Instantiator but does not invoke it's
constructor.  Therefore, I manually invoke the "hookInit" that I created
inside _Instantiator.


_always_comb.py:  Same as _always.py.  Manually invoke the "hookInit" that
I created inside _Instantiator.


_always_seq.py:  Same as _always.py...


_extractHierarchy.py:  Tweak _HierExtr and _Instance to preserve hookData
placed via hookInit from _Instantiator.  I also tweaked _UserCode and
_addUserCode to preserve hookData, but I haven't tried any user-defined
Verilog (or VHDL) to test this.  TODO: If anyone has user-defined code, try
putting it in a partition (perhaps easiest via "with"), and tell me if it
is placed in the correct Verilog module on output.


_contextdecorator.py:  New file, from
http://pypi.python.org/pypi/contextdecorator to support both "with
partition(...) ..." and "@partition(...)" mechanisms.


_partition.py:  New file, and is actually pretty short.  It defines
"partition" and "get_current_partition" API, and inject the partition tags
in the relevant objects such as @always, @always_comb, @always_seq.


conversion/_analyze.py:  Preserve hookData in _analyzeGens.  And
in _AnalyzeVisitor, I added tree.inmems to mirror record keeping of
tree.outmems, so that I can verify that memories are only read from the
same partition in which they're driven.


conversion/_toVerilog.py:  Thar be dragons.  Here is where I added
toVerilog.partitioned_output, to invoke my alternative Verilog output code
contained in _write_partitioned_output(...).  TODO:  I should
move _write_partitioned_output and its supporting entourage to another
file.  (SigMapping,
lookup_id, distill_signals_and_memories, _apply_renames,
_generate_partitions, _partition2name, _make_unique_names)







On Sun, Sep 29, 2013 at 8:25 PM, Keerthan jai.c <jck...@gm...>wrote:

> David,
>
> Just a heads up, the 0.9 dev branch on bitbucket has quite a few changes
> in the conversion area(MEP 107). You might want to check whether these
> patches apply on the new branch.
>
>
> On Sun, Sep 29, 2013 at 2:21 PM, David Holl <da...@ad...> wrote:
>
>> There was an old thread regarding MEP 110, but I've come up with an
>> alternative "solution", and I'm looking for your comments.  (call it MEP
>> 110-alt?)
>>
>> tl;dr --- If you're interested,
>>
>> Introduction:  Same as MEP 110.  (Especially for floor planning...)
>> -----------------------
>> Analysis:  Same as MEP 110.  (Yep, everything gets flattened by
>> default...)
>> -----------------------
>> Proposed Solution:  (with implementation)
>>
>> I did not follow the proposed solution in MEP 110, because that would've
>> required that I change [any of] my components to be "top-module
>> convertible".  (I'm using a form of signal containers --similar to MEP
>> 107-- which which would've required many changes (in my code) to achieve
>> "top-module convertibility".)
>>
>> What I did was I started from the flattened hierarchy in
>> conversion/_toVerilog.py, but instead of calling the usual
>>  _writeSigDecls, _convertGens, I wrote my own convoluted output code:
>>     ...
>>             if self.partitioned_output:
>>                 _write_partitioned_output(self.partitioned_output, vfile,
>> vpath, self.timescale, h, intf, siglist, memlist, genlist, doc)
>>             else:
>>                 _writeFileHeader(vfile, vpath, self.timescale)
>>                 _writeModuleHeader(vfile, intf, doc)
>>                 _writeSigDecls(vfile, intf, siglist, memlist)
>>                 _convertGens(genlist, vfile)
>>                 _writeModuleFooter(vfile)
>>     ...
>>
>>
>> To activate this alternative output, just
>> set toVerilog.partitioned_output=True:
>>  toVerilog.name='test_partition'
>> toVerilog.partitioned_output=True  # the default is False
>>  toVerilog(_test_top, clk, AAA, BBB, ZZZ, TTT, VVV)
>>
>> Without any other changes to a hardware description, this alternative
>> code will produce identical, flattened output. (right down to the arbitrary
>> ordering of signals and generators...)
>>
>> However, you may now group any related logic together, as needed.  I call
>> a group of logic a "partition", and these partitions may be nested.
>>  Instead of the word "partition" you could call it a "module" instead,
>> since the concept is almost identical to Verilog module's.  In the
>> following description, it will appear that I'm flipping between "partition"
>> and "module" when really, I'm describing how that partitions in python
>> impact the created Verilog modules.  (but for simplistic purposes, since I
>> guarantee each "partition" will create a unique Verilog "module", feel free
>> to call em' what you want...)
>>
>> There are 2 ways to assign related hardware into partitions.
>> -----------------------
>> 1) The first is via python's "with" statement.  I'll show a few examples:
>>
>> example 1.a) A simple partition...
>> from myhdl import Signal, partition, ...
>>
>>  @always(clk.posedge)
>> def clgc():
>> c.next = a + b
>>
>> with partition('A'):
>> @always(clk.posedge)
>>  def clgd():
>>  d.next = c + e
>>
>> @always(clk.posedge)
>> def clge():
>>   e.next = a * b
>>
>>  @always(clk.posedge)
>> def clgf():
>>  f.next = more stuff...
>>
>>
>> What "partition" does is annotate all generators (and shadow signals...)
>> with your text label.  Then the "partitioned-output" code uses these
>> annotations move related logic into a Verilog module, and instantiate that
>> module from the top-level module.  Module ports are automatically created
>> for any signals that need to cross between modules.
>>
>> This code still creates only 1 output .v file containing all modules.  (I
>> didn't care if it was 1 file or multiple files, but for me, 1 file was
>> "easier" to move/copy around in my opinion.)
>>
>> The top module will include all of the signals, except for "e" ---
>> because in this code, e is only used within partition A.  Therefore "e"
>> will appear as a driven "reg" within a submodule called A. (and that
>> submodule "A" is instantiated in the top level as "part_A"...)
>>
>>
>>
>> example 1.b) Partitions may be nested.
>>
>> @always(clk.posedge)
>>  def clgc():
>> c.next = a + b
>>
>>  with partition('A'):
>> @always(clk.posedge)
>>  def clgd():
>>  d.next = c + e
>>
>>  with partition('B'):
>>  @always(clk.posedge)
>> def clge():
>>   e.next = a * b
>>
>> @always(clk.posedge)
>>  def clgf():
>>   f.next = more stuff...
>>
>>
>> This code will create 3 modules total:  the top module, a module called
>> "A", and a module called "A_B".
>>
>>
>> example 1.c) Partitions having the same name will be renamed to stay
>> unique!
>>
>> If you want related code to be grouped into a partition, all of that code
>> must be contained in the same python "with" block.  For example:
>>
>> from myhdl import Signal, partition, ...
>>
>> @always(clk.posedge)
>> def clgc():
>>  c.next = a + b
>>
>> with partition('A'):  # This will be renamed to A0
>>  @always(clk.posedge)
>> def clgd():
>>   d.next = c + e
>>
>>  with partition('A'):  # This will be renamed to A1
>> @always(clk.posedge)
>>  def clge():
>>  e.next = a * b
>>
>>
>> Rationale:  When you start nesting partitions, partition names are
>> guaranteed to produce unique module instances and not accidentally group
>> unrelated logic together.
>>
>>
>> example 1.d) Partitions are nestable across your python function calls,
>> and renaming will happen to keep sub
>>
>> For example, unrelated nested partitions may use the same name without
>> fear of their logic being accidentally grouped together.
>>
>> from myhdl import Signal, partition, ...
>>
>> def my_module_alice(...):
>>  # ...
>> with partition('SM'):
>>  # This happens to enclose a state machine.
>>   # The name SM isn't descriptive enough, but if you
>>   # want descriptive Verilog output, then you should
>>   # use descriptive names.
>>   # ...
>>  # ...
>>  return instances()
>>
>>  def my_module_bob(...):
>>  # ...
>>  with partition('SM'):# This encloses an unrelated state machine.
>>   # ...
>>   # ...
>>  return instances()
>>
>> def top(...):
>>   @always(clk.posedge)
>> def clgc():
>>   c.next = a + b
>>
>> with partition('CRYPTO'):
>>   alice_inst = alice(...)
>>   bob0_inst = bob(...)
>>  bob1_inst = bob(...)
>>
>> with partition('CRYPTO_SM2'):
>>  # I made this CRYPTO_SM2 name, just to be a jerk.
>>   @always(clk.posedge)
>> def clgd():
>>   d.next = c + e
>>
>>
>> This code would produce 6 instantiated modules in the generated Verilog
>> output:
>> module top(...);
>>  module CRYPTO(...);
>> module CRYPTO_SM0(...); // this is from alice_inst
>> module CRYPTO_SM1(...); // this is from bob0_inst
>>  module CRYPTO_SM2(...); // this is from bob1_inst
>> module CRYPTO_SM2_(...); // renamed CRYPTO_SM2 to not conflict with
>> CRYPT_SM2 from bob1_inst
>>
>> Admittedly, the generated names CRYPTO_SM... in this example aren't very
>> clear, but the point here is that your python functions (like alice() and
>> bob()) have a guaranteed means to group logic into a Verilog module without
>> accidental commingling of unrelated from other modules.
>>
>> The "top" module instantates "CRYPTO" and "CRYPTO_SM2_".  Then "CRYPTO"
>> instantiates "CRYPTO_SM0", "CRYPTO_SM1", and "CRYPTO_SM2".
>>
>>
>> example 1.e) Partitions support Verilog 2001 attributes  (and if someone
>> wants to add VHDL support...)
>>
>>  with partition('A') as this_part:  # This will be renamed to A0
>>  this_part.attr_add('KEEP_HIERARCHY', 'TRUE')
>>  this_part.attr_add('DONT_TOUCH', 'TRUE')
>>
>>  @always(clk.posedge)
>> def clgd():
>>  d.next = c + e
>>
>> This will produce output that instantiates module A as:
>>
>> (* KEEP_HIERARCHY="TRUE", DONT_TOUCH="TRUE" *) part_A A(clk, c, e, d);
>>
>> // The generated module for instance A is called "part_A".
>>
>> (MyHDL also has a nifty feature to include doc strings as embedded
>> comments.  I have some back-end support for attaching a string to a
>> partition, to be output as a comment in the code, but I haven't turned it
>> on yet.)
>>
>> -----------------------
>> 2) The second way to group logic into a partition is with a python
>> decorator.
>>
>> example 2)  Using decorators in addition to "with" blocks.
>>
>> The decorator syntax uses the same rules as the with-blocks.  (All
>> partitions will be unique, and possibly suffixed with a number to maintain
>> uniqueness in the Verilog code.  Partitions may be nested.  Use descriptive
>> names if you want descriptive output.)
>>
>> Here, I'll intermix the decorator and with syntax:
>>
>> def my_module_alice(...):
>>  # ...
>> with partition('SM'):
>>  # This happens to enclose a state machine.
>>   # The name SM isn't descriptive enough, but if you
>>   # want descriptive Verilog output, then you should
>>   # use descriptive names.
>>   # ...
>>  # ...
>>  return instances()
>>
>>  @partition('AES') # <---- partition a la decorator.
>>  def my_module_bob(...):
>>  # ...
>>  with partition('SM'):# This encloses an unrelated state machine.
>>   # ...
>>  # ...
>>  return instances()
>>
>> def top(...):
>>   @always(clk.posedge)
>> def clgc():
>>   c.next = a + b
>>
>> with partition('CRYPTO'):
>>   alice_inst = alice(...)
>>   bob0_inst = bob(...)
>>  bob1_inst = bob(...)
>>
>> with partition('CRYPTO_SM2'):
>>  # I made this CRYPTO_SM2 name, just to be a jerk.
>>   @always(clk.posedge)
>>  def clgd():
>>   d.next = c + e
>>
>> The above code will generate ~8 modules:
>> module top(...);
>> module CRYPTO(...);
>>  module CRYPTO_SM(...); // this is from alice_inst
>>  module CRYPTO_AES0(...); // decorator on bob()
>>  module CRYPTO_AES0_SM(...); // this is from bob0_inst
>>  module CRYPTO_AES1(...); // decorator on bob()
>>  module CRYPTO_AES1_SM(...); // this is from bob1_inst
>>  module CRYPTO_SM2(...); // renamed CRYPTO_SM2 to not conflict with
>> CRYPT.SM from bob1_inst
>>
>>
>> ----------------------- END OF EXAMPLES
>>
>> Limitations:  I hope there aren't any.  I put enough care to accomodate
>> ConcatSignal's, _SliceSignal's, const [undriven] _Signal's, etc...  And if
>> a signal gets pushed into a sub-partition where it is both driven and read,
>> as well as output from that sub-partition, the generator will automatically
>> make a separate wire just for the output port, that is driven from your
>> signal.  (This prevents "Output port is read internally..." warnings for
>> partition-generated modules.  I don't do this for the top-level though, to
>> preserve the same interface as current MyHDL output.)
>>
>> Component Conversion:  No additional restrictions beyond that of MyHDL.
>>  There is still the same requirement on your top-level to be "top-level
>> convertible", but there is flexibility for your lower levels.
>>
>> Component Parameters:  What parameters?  All of your parameters were
>> already digested by python+MyHDL magic, and I'm operating on the flattened
>> hierarchy.  If you invoke the same sub-partition (sub-module) multiple
>> times, it will be replicated in the output, but every time it appears, it
>> will use the globally-unique signal names that MyHDL has already assigned.
>>  (ie, I don't change your signal names.)
>>
>> Any drawbacks:
>> This may/will produce larger output code than the proposed solution in
>> MEP 110.  The existing MEP 110 solution will create 1 copy of each module,
>> and instantiate it multiple times (as long as each instantiation shares the
>> same port signal widths, etc...).  In contrast, this "110.alt" solution
>> will create a whole copy of a module every time it is instantiated.  (But
>> each copy inherits the unique signal names assigned by MyHDL's flattening
>> algorithm.)  However, the larger output size from this "110.alt" isn't that
>> much larger than MyHDL's native flat output.  (depending on your signal
>> usage...)
>>
>> Benefits:
>> You can have hierarchical output with minimal changes to your code,
>> and you control exactly how your logic is partitioned into the hierarchy.
>>  Just sprinkle a few @partition(...) or "with partition(...):" statements
>> wherever you need in your design.
>>
>> Known pitfalls:  I have not tried or tested TristateSignal's for "inout"
>> ports, and I've only tested on a very narrow subset of MyHDL's feature set.
>>  (I was was already bitten by ConcatSignal's, _SliceSignal's, and lists of
>> signals accidentally being inferred as memories...  However, I expect these
>> areas to work as expected now.)
>>
>>
>> But if anyone is interested, here's the code, and I'm open for feedback.
>>
>> I attached patches against myhdl-0.8.tar.gz available from
>> https://sourceforge.net/projects/myhdl/files/myhdl/0.8/
>>
>> Of the 2 patches, myhdl-0.8-fix_modbv.patch is my poor-man's fix for
>> modbv, because I needed modbv.  This patch is most likely obsoleted by
>> Jan's real fix in the repo.
>>
>> The other patch (myhdl-0.8-partitions.patch) contains my crude first
>> draft supporting the described hierarchy.  Yes, this code is ugly.  Don't
>> look at it yet.  It is a quick mock-up of some functionality I really
>> wanted.  But give it a try if you're interested and let me know what it
>> breaks on...  Or if you really want to dive into the code, feel free to do
>> so.
>>
>> On my systems, I apply & install these patches as follows:  (gunzip the
>> .patch files first)
>>
>> gunzip myhdl-0.8-fix_modbv.patch.gz
>> gunzip myhdl-0.8-partitions.patch.gz
>> tar zxf myhdl-0.8.tar.gz \
>> && cd myhdl-0.8 \
>> && patch -p1 < ../myhdl-0.8-fix_modbv.patch \
>>  && patch -p1 < ../myhdl-0.8-partitions.patch \
>> && python setup.py build \
>>  && python setup.py install --user -O2 \
>> && cd .. \
>>  && rm -rf myhdl-0.8
>>
>> This installs myhdl into python's standard location for my home
>> directory, but feel free to leave off --user to install to the main system
>> path, or wherever else you want.
>>
>> To test drive the functionality, try running test_partition.py.  Set
>> toVerilog.partitioned_output=False
>> to see the normal output, and
>> toVerilog.partitioned_output=True
>> to see the partitioned output.  And if you are curious, try
>>  toVerilog.partitioned_output="I promise I will never die."
>> to enable the partitioned output code, but tell it to ignore the
>> partition annotations.
>>
>>
>> (typos and all, I'm hitting send...)
>>
>> - David
>>
>>
>>
>> On Wed, May 1, 2013 at 6:44 PM, Oscar Daniel Diaz <osc...@gm...>wrote:
>>
>>> El Wed, 01 May 2013 10:55:55 +0200
>>> Jan Decaluwe <ja...@ja...> escribió:
>>>
>>> > On 04/29/2013 05:25 PM, Oscar Daniel Diaz wrote:
>>> >
>>> > > Given that the converter is called recursively several times,
>>> > > multiple files are generated and its contents are used for component
>>> > > declaration (VHDL). Recursive calls must avoid file replication (for
>>> > > VHDL case means avoid multiple package file generation). This is
>>> > > done by keeping generated code in memory before pushing to files.
>>> > >
>>> > > Attribute "no_component_files" :
>>> > >   * False : all components code is saved to disk as soon as
>>> > > possible.
>>> > >   * True : discard components code, only save top-module file.
>>> >
>>> > Not sure what is meant here exactly. Why would we need component
>>> > declarations, as opposed to simply using direct instantation?
>>> > For a case like this, this seems much simpler and less verbose
>>> > to me.
>>>
>>> Since keep hierarchy conversion requires multiple files to generate
>>> (top-module and components to instantiate), this flag prevents
>>> generation of those components. Suppose you make some changes only in
>>> the top-module, this flag allows to only generate top-module file.
>>>
>>> When I change from VHDL '87 syntax (doing recursive calls to get
>>> component declaration) to '93 syntax, that flag would prevent
>>> those recursive calls.
>>>
>>> --
>>> Oscar Díaz
>>> Key Fingerprint = 904B 306C C3C2 7487 650B  BFAC EDA2 B702 90E9 9964
>>> gpg --keyserver subkeys.pgp.net --recv-keys 90E99964
>>>
>>>
>>> _______________________________________________
>>> myhdl-list mailing list
>>> myh...@li...
>>> https://lists.sourceforge.net/lists/listinfo/myhdl-list
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
>>
>> _______________________________________________
>> myhdl-list mailing list
>> myh...@li...
>> https://lists.sourceforge.net/lists/listinfo/myhdl-list
>>
>>
>
>
> --
> have a nice day
> -jck
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
> _______________________________________________
> myhdl-list mailing list
> myh...@li...
> https://lists.sourceforge.net/lists/listinfo/myhdl-list
>
>