Re: [myhdl-list] Starting over

[Jan D. <ja...@ja...>]

Bob:

Now I think we are getting somewhere. A couple of things:

First, there is some confusion about what Migen exactly means.
My fault partially.

Apparently Migen packages both a HDL, FHDL, and some IP
blocks. When I criticize Migen, I'm really addressing FHDL,
because that's my interest in comparing with MyHDL. The
problem is that it is not packaged separately.

I don't think it's a good idea to package IP blocks together
with an HDL, and I will not do that for MyHDL. It's a totally
different, and much bigger and diverse task.

However, you do point to an important weakness of the MyHDL
ecosystem: the lack of strong open source IP blocks. This is
somewhat frustrating, because I think MyHDL is the ideal
platform for such developments. There is no reason why it
should be less powerful than Migen for this, to the contrary
(because of the more powerful underlying HDL).

I add that I personally have not plans in this direction. You
cannot do it all. The Migen developers apparently included
exactly the IP that they need for their specific purposes.
I hope to see MyHDL IP in a variety of domains, including
ones that I know nothing about.

The main conclusion from your post is that I should do a
better job in upfront expectation management. The very next
thing I'm going to do is to write a page "What MyHDL is not".

Jan


On 04/22/2012 08:53 PM, Bob Cunningham wrote:
> Thank-you Jan for your careful and caring response.  It is clear I
> haven't gone far enough down my own learning path to make
> well-founded criticisms of HDL tools or approaches.  While I continue
> to closely monitor this list and developments in the general world of
> "alternative" HDLs, a recent job change has temporarily eliminated my
> free time to pursue learning HDLs: My FPGA development board is
> gathering dust on my workbench.  Still, when I finally do resume my
> efforts, I hope to have a better path to pursue.
>
> Rather than attempt to respond to your points individually, please
> permit me instead to step back and restate my goals and describe how
> they have evolved since encountering MyHDL and Migen.
>
> As an embedded real-time engineer specializing primarily in software,
> but with proven skills in system design and high-level circuit design
> (mainly processor and peripheral chip selection and integration), my
> greatest goal is to minimize the amount of new software in the
> products I create.  Statistically, nothing adds bugs to a system
> faster, and delays product shipping longer, than adding a single new
> line of code.  The buggiest code tends to occur at the lowest levels:
> At the functional level, the error rate for assembler is a multiple
> of that for C which in turn is a multiple of that for Python.
>
> It's not just about levels of abstraction, though that is certainly a
> vital factor.  It also concerns the availability of proven code, and
> how easily that code can be integrated onto the current project.
> Part of the issue is how easy or difficult it is to create portable
> and reusable code on any system, the rest concerns how to make use of
> that code.  I love projects where most of my effort is spent writing
> "glue" code to integrate known-good libraries to create a new
> application.  My work as an engineer then becomes finding that good
> code, and selecting from among the candidates the code that best
> meets my needs and will be easiest to integrate.
>
> That said, what I love most is writing code from scratch,
> implementing something important that has never existed before.  But
> that's not what gets products out the door:  Professionally, I must
> hold my own code authoring skills and desires at bay, and use the
> rest of my skills to assess what's best for the system and product as
> a whole.  It's kind of schizophrenic:  I ship better systems sooner
> when I keep my own coding skills as a last resort, and prioritize
> integrating the work of others.  But I always yearn for projects
> where I get to do something truly new and innovative.
>
> I view the FPGA as an ideal canvas for an embedded engineer, where I
> can work more closely with the EEs while simultaneously using using
> my system design and integration skills to further reduce the
> software needed to produce a product.
>
> The first FPGA skill I'd most like to have to be productive would be
> the ability to write glue between the FPGA and devices in the outside
> world, and between IP cores within the FPGA.  I became very excited
> when I saw that Migen included the Wishbone interface as a
> pre-integrated macro: There are many cores over at OpenCores that use
> the Wishbone interface, but getting them to talk to each other
> required a level of skill that was way beyond my knowledge.
>
> Why would I need to know how to implement a Wishbone interface?  What
> I need to know is how to design and test a system that uses
> components integrated with standard interfaces, and those skills are
> readily transferred from my embedded experience, where I have had to
> test and integrate many hardware systems and components.  I like
> using logic analyzers and o'scopes far more than I like using
> software debuggers!
>
> I suppose this isn't really an HDL issue:  It's much higher than
> that.  I suppose it's more of a tool-level issue with language
> implications.
>
> When I finally do start to write my own complex circuits from scratch
> (something I greatly look forward to, and expect to have lots of fun
> doing), I'll want to use a language (or languages) that are
> appropriate to the task at hand, much as I use assembler, C and
> Python to create embedded software.
>
> However, identifying the most appropriate language can be difficult,
> and may even be a needless effort if the highest-level languages can
> "play nice" at the lowest level.  Permit me to share a case in
> point:
>
> Several years ago I had to develop an instrument based on a complex
> sensor system that used inherently noisy sensors from which we needed
> to extract "soft" real-time performance.  The statistics alone were
> quite daunting, and I decided to prototype the system in Python so I
> could use Numpy and related libraries to quickly develop the
> algorithms I needed.  The prototype was a success, but it had one
> interesting characteristic: It met our performance sped when running
> on the desktop, and was only a factor of 2-3 slower than our minimum
> required performance spec when running on our intended target system.
> Surprising performance for a prototype in an interpreted language.
>
> I profiled the code and found that the Python numeric and statistical
> libraries were wicked-fast: They were more than capable of doing all
> the math needed well within our timing requirements.  That meant the
> slow performance was due to the code I had written to interface to
> the sensors and prepare their data for use by the fast libraries, and
> passing data between those libraries.  I first moved as much
> functionality as possible into the libraries, which yielded an
> immediate 25% speed improvement.  Next, I stumbled across Psyco
> (these were pre-PyPy days), and used it to more than double the
> performance of the non-library code I had written.
>
> That left me tantalizingly close to meeting my timing requirements,
> but I was still missing them by 30%.  I had never before considered
> using Python to implement a near-real-time embedded system.  Not only
> that, I also had never shipped Linux in a delivered system: I had
> always been driven to minimize system power, which meant small
> processors, which in turn meant running no code that didn't directly
> support the end result, which in turn meant often running with no OS,
> or at most an RTOS.  For this system I had a much more generous power
> envelope due to the generous maximum weight my battery-powered
> instrument could have.  If I went with a Linux platform, I'd need to
> double my power budget, which meant doubling the current rating and
> capacity of the batteries: Switching from NiMh to Li-Ion permitted me
> to get the power I needed with only a modest increase in system
> weight.
>
> But how to obtain that last bit of performance I needed?  I was
> already using the fastest CPU I could get that met our packaging
> requirements (no fan, no extensive or expensive passive heatsinks,
> which at the time meant a ULV Celeron).  A final round of profiling
> indicated my interface code was spending lots of time reading and
> converting data:  Each data element was causing individual library
> calls.  Implementing larger buffers helped, but still didn't get me
> close to the time I needed: Servicing the hardware was the problem.
> The final speed boost came when the buffers were moved from software
> to hardware, so each interface access returned 1000x more data, a
> design change that also improved the timing between the individual
> sensors, which in turn simplified the statistics needed to proce ss
> the data.
>
> Sorry for the length of that story, but that experience has affected
> all the work I've done since, and also affects my expectations when
> it comes to FPGA development.  I want my FPGA development to
> integrate nicely into the above process, to be a powerful tool that
> permits me not just to engineer a fully-specified circuit into a
> fully-specified target, but also to experiment with integrating
> different components, to move solutions or parts of the problem from
> one domain to another, to quickly make use of the work of others, and
> be able to create my own when needed.
>
> The key element that must be present to facilitate such flexibility
> is the ability to implement common aspects of interfaces across
> multiple domains, to switch interfaces while preserving
> functionality, to be able to work with gates, busses, processors,
> instructions, libraries, languages and whatever else is needed to
> permit a given slice of functionality to be implemented and accessed
> in its best and most effective domain.  I want everything from simple
> FIFOs to complex processor interfaces to be available to me in
> multiple domains without having to create them from scratch each
> time.
>
> And THAT is what initially attracted me to MyHDL:  If I can use
> Python to move software into hardware, and interface with it from the
> Python domain, why, I could become a system development god!  And the
> advent of PyPy makes that approach even more tractable.
>
> I soon learned that such power is not the goal of MyHDL:  The goal of
> MyHDL is to use Python to make a incrementally better flavor Verilog
> or VHDL, a relatively small but significant evolutionary step
> compared to the transformative change I was dreaming of.
>
> I desire to climb up and skip between levels of abstraction and
> implementation domains that are almost orthogonal to what MyHDL
> pursues.  MyHDL may prove to be a piece of the final process, but it
> does not, and perhaps cannot, encompass that process on its own.
>
> Hence my excitement with Migen:  Even in its embryonic, incomplete,
> and flawed initial state, it clearly seeks to bridge abstractions in
> ways alien to MyHDL, to make interfaces an integral part of the
> package and process, rather than something to code to at a low level.
> In this respect, MyHDL feels more like C, and Migen aims to be Python
> with its powerful libraries being made available to higher-level
> code.
>
> Again, I'm only mapping from domains I know to a domain I hope to
> enter and in which I hope become proficient and productive.  I'm
> certain my mapping has flaws and gaps owing to my lack of general HDL
> knowledge.  But I suspect this very lack may enable me to conceive of
> and pursue a path that may yield a greater whole.
>
> MyHDL and Migen both seem to me to be stepping stones in a stream I
> wish to see crossed by a 4-lane bridge.  And arguments between and
> about them seem like arguments about C vs. Python:  It's about the
> solution path and the power of the process, not about any particular
> tool!  It's about how tools interact and combine, not about how or
> why they differ or overlap.
>
>
> And if Python/PyPy doesn't evolve more quickly, it may get devoured
> by Julia, an interpreted/JIT language which abstracts into areas
> Python presently supports badly (such as multiprocessing and
> coroutines), with speeds PyPy has yet to attain.  It may be that the
> concepts embodied in both MyHDL and Migen could eventually see more
> effective and more flexible implementations in the Julia ecosystem.
> Python, too, is just one tool in a procession of tools over time.
>
>
> -BobC
>
>
> On 04/22/2012 03:25 AM, Jan Decaluwe wrote:
>> I still want to take the time to clarify my position on the many
>> issues raised in this post.
>>
>> On 03/17/2012 09:20 PM, Bob Cunningham wrote:
>>> On 03/16/2012 02:03 PM, Jan Decaluwe wrote:
>>>> My conclusion is that Migen found an easy target in you, as a
>>>> newbie, to confuse you. It made you think it can be used for
>>>> serious design work.
>>> Sorry, Jan.  If I have to be "confused" to play with my FPGA,
>>> then so be it.  I'm very "serious" about being able to play with
>>> my FPGA!
>>>
>>> Your statement has an obvious implicit context: To me, you are
>>> shouting, "MyHDL is for Serious Designers Only!  Newbies and
>>> Pragmatists should Go Away!"
>>>
>>> If that's what you are saying, then please be direct: Don't
>>> attack Migen for being what it was intentionally created to be,
>>> or for being something MyHDL is not intended to be.  Are you
>>> upset about Migen existing, or that there is an audience MyHDL
>>> can't help as well as Migen may be able to?
>> I am disturbed by the suggestion that my critique on Migen is based
>> on anything else than a purely technical assessment.
>>
>> Let me be clear. I don't like Mr. Bourdeauducq's attitude for one
>> bit. But do you think that would be a reason for me to ignore any
>> good idea that he might come up with? Of course not. I am not a
>> masochist.
>>
>> It is quite simple. During my career, I have found that when you
>> think you have seen the worst in HDL-based design, it always gets
>> worse. To date, Migen is the worst that I have seen. But to
>> understand why I am saying this, you have to be prepared to follow
>> my technical arguments and to engage in technical discussions. I
>> have made a few starts, but I certainly was not done yet. However,
>> I see close to zero enthousiasm to continue such discussions.
>>
>> I am therefore frustrated by the fact that I hear all kinds of
>> opinions and suggestions to "merge" but that whenever things get a
>> little technical then the "I am a beginner" umbrella opens.
>>
>> Migen is not my problem. It will disappear in the milky mist of HDL
>> history, just like the many HDLs based on the same flawed paradigm.
>> I am addressing it simply because misleading posts about it appear
>> on this newsgroup.
>>
>> What I am really targetting instead is the conventional wisdom in
>> mainstream HDL design, which often has it all wrong.
>>
>>> If you'd rather beginners like myself go elsewhere, just say so.
>> MyHDL welcomes beginners. It is the first item on "Why MyHDL":
>>
>> http://myhdl.org/doku.php/why#you_are_new_to_digital_hardware_design
>>
>>
>>
In fact, I put most of my hopes on beginners, as they have not
>> yet been brainwashed by the conventional wisdom.
>>
>>> Remember, Migen wasn't created to be useful to beginners: It was
>>> created by an experienced FPGA designer concerned about
>>> practicality and productivity for a specific Open Source hardware
>>> project.  It
>> I am not impressed by "arguments from experience". The
>> conventional wisdom that I am targetting was created by experienced
>> designers, mostly Verilog ones. Experience can lead to conservatism
>> and can become a hindrance to clear thinking.
>>
>>> simply happened that some things became a bit clearer to me
>>> after looking at Migen and the problems it was created to
>>> address.
>> Was it because of Migen or simply because you were spending more
>> time on the problem?
>>
>>> Whatever Migen leaves out may have been what was getting in my
>>> way!
>> I find that strange. I understand that you have a lot of experience
>> with embedded software. Therefore, you must know procedural
>> software techniques very well. That is exactly what Migen leaves
>> out. What it leaves is low-level concurrency at the statement
>> level, which must be new to you. And now you suggest that the
>> obstacle is exactly that what you are most familiar with. Beats
>> me.
>>
>>> I'm actually quite lazy: What is the *least* I need to know to
>>> make useful digital designs *now*?
>> No secrets here. The first sentence of "Why MyHDL" warns you:
>> "There's a lot to learn and it will be hard work". Therefore, if
>> you are intellectually lazy (not prepared to learn new things even
>> when they will save you lots of time and effort later on), MyHDL or
>> HDL-based design is not for you.
>>
>> MyHDL is for those who are lazy in the good engineering sense,
>> wanting to accomplish more with less effort eventually.
>>
>>> I'm a beginner:  Though I'd love to someday be able to design
>>> circuits like Shakespeare wrote sonnets, I'd be more than happy
>>> today if I were able to work at the level of "Green Eggs and
>>> Ham", a true masterpiece written with an absolute minimum of
>>> linguistic complexity.
>> Come on, let's keep some perspective here. It's not *that*
>> difficult or complex either. And there is a cookbook that shows you
>> the way.
>>
>>>> When Migen claims that the "event-driven" paradigm is too
>>>> general, what it really dumps is procedural support in your HDL
>>>> descriptions - the interesting stuff.
>>> What's "interesting" to you can be a frustrating block for a
>>> newbie. I hope to one day also be very interested in those
>>> aspects of MyHDL, but it seems to have little to do with what I
>>> want to get done today,
>> I don't understand. Your specification seems very extensive and
>> ambitious. It would seem that you have a big need for raising the
>> abstaction level as high as possible, and for an easy path to
>> strong verification.
>>
>>> which is to find the simplest path to get basic circuits out of
>>> my mind and in to my FPGA.  Then use them as building-blocks in
>>> my hobby projects.
>> There is a broad concensus about the "building blocks" paradigm in
>> hardware design. That is really not the issue. The issue is what
>> the abstraction level of the building blocks should be.
>>
>>> I am a MyHDL fan.  Unfortunately, I simply remain unable to use
>>> MyHDL to accomplish my own immediate hobby needs.  That does not
>>> indicate any flaw in MyHDL, merely the extent my own limitations.
>>> Do not be surprised that I am interested in any tool that helps
>>> me circumvent those limitations!
>>>
>>> I actually *like* how Migen slices and dices the process of FPGA
>>> design: The parts that are left out are ones I doubt newbies like
>>> me would notice, much less care about, until confronted with
>>> unusually challenging designs.  I suspect Sebastien would agree
>>> with much of your analysis, the proper response being: "So
>>> What?"
>> Suppose that  I teach a class to newbies in embedded software
>> design based on assembler. Would any of the newbies, except for
>> the rare genius, miss the capabilities of say, C?  Does this prove
>> that teaching assembler was a good choice?
>>
>>> It's not about theoretical power or completeness: It's about
>>> barriers to entry.  It's not about what I can do in 5 years, but
>>> about what I can do in 5 weeks.  Migen is primarily about
>>> pragmatism and productivity, making powerful circuits quickly and
>>> easily, and far less about expert capabilities, theoretical
>>> purity or even consistency.
>> Again, I find this strange. I understand that you have not been
>> successful with MyHDL. However, as I understand it you have not
>> been successful with Migen either. So what is your defense based
>> upon? Of course, we are about 5 weeks further now :-)
>>
>> More to the point.
>>
>> Barriers to entry - ok, but what is the task? I told you that I
>> believe the main problem in HDL-based design is verification, and
>> how MyHDL (unlike Migen) helps you by the fact that you can use the
>> same modelling paradigm for high-level models and test benches as
>> for synthesizable logic.
>>
>> You seemed surprized, which I found suprizing in turn. Is it so
>> different in software? Really, getting those gates into an FPGA is
>> the easy part. The difficult part is getting them to work
>> properly.
>>
>> You will have noticed that Mr. Bourdeauducq made an error in the
>> first simple "one-liner" circuit that I presented to him, as if he
>> wanted to prove my point. Of course, the reason is not
>> incompetence, but simply that he did not verify his design.
>>
>> There is a pattern however. Mr. Bourdeauducq cried foul because I
>> didn't accept his "simple patches". What he ignored, and continued
>> to ignore despite my insistence is that they broke MyHDL. Perhaps
>> Mr. Bourdeaducq considers verification a "detail".
>>
>> Well, I don't. Verification is the problem. The reason why I think
>> the abstraction level of synthesizable logic should be as high as
>> possible, is because that leaves more time for verification.
>>
>>> I seek tools that will help me do what I want to get done today,
>>> and so far Migen seems like it will be most useful.  Tomorrow
>>> will likely require additional tools, and I *absolutely* expect
>>> (and want) MyHDL to the first of those tools.  It is not an
>>> either-or proposition: I want Migen *and* MyHDL.  I realize MyHDL
>>> will take more time to master, and I'm willing to commit that
>>> time.  But I also want to create something sooner, rather than
>>> later.  And I believe that 100% of what I learn using Migen will
>>> later prove useful with MyHDL.  I believe using Migen will keep
>>> me motivated toward learning MyHDL.
>> Sounds good, but I think it is cheap talk.
>>
>> Most of Migen's technical choices, starting with its basic
>> paradigm, are almost the opposite of MyHDL's. As a result,
>> verification is not addressed, and it forces you to think at an
>> artificially low level for synthesizable logic. What good can one
>> learn from that?
>>
>>> Right next to me I have a Spartan 3E-500 that contains nothing of
>>> my own design.  That must change!
>> Perhaps you are too ambitious.
>>
>> In your shoes, I would start as follows:
>>
>> * isolate a simple function out of your spec * try to concentrate
>> on what it does, instead of how it should be implemented * write
>> that behavior in a (clocked) MyHDL process or processes * also
>> describe it in high-level python, and use that in a unit-test to
>> verify * experiment with synthesis to see whether it works and the
>> result is acceptable * iterate with the synthesizable description
>> as necessary
>>
>>
>
> ------------------------------------------------------------------------------
>
>
For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You. Monitor Your
> Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2


-- 
Jan Decaluwe - Resources bvba - http://www.jandecaluwe.com
     Python as a HDL: http://www.myhdl.org
     VHDL development, the modern way: http://www.sigasi.com
     World-class digital design: http://www.easics.com