Extending Tcl
blacksqr
7 years ago
120
55
http://wiki.tcl.tk/6276
davidw7 years ago
Tcl's C API is very nice, and makes it easy to either integrate Tcl into a larger C app, or write bits and pieces of C as Tcl commands.

I helped work on the 2nd edition of this book, which I think is still fairly relevant: https://amzn.to/2IB9OyR

pjmlpdavidw7 years ago
Writing Tcl extensions was one of the few times I actually enjoyed using C.

It is probably one of the best languages to extend. Sadly it is also one of the slowest.

hedorapjmlp7 years ago
Years ago, I had to port a tcl/tk ui to java swing. I had source code to copy off of, but after two or three rewrites, the Java thing was still at least 100x slower than the tcl thing (granted, this was in the 1.1/1.2 days, but still...)
gaiuspjmlp7 years ago
Really, it flies compared to Ruby...
sreangaius7 years ago
Is that right ? I have never used Ruby. I should have my expectations on running time in check in case I do try Ruby out -- I have been meaning to.
gaiussrean7 years ago
I don’t dislike Ruby - you can do some very clever things in very little code with it. But it is far from high-performance!

Early Tcl was slow - but it was being compared to pure C. Recent Tcl is as snappy as any HLL.

bchgaius7 years ago
Early (pre-8.0) Tcl was also literally string-processing the scripts. 8.0 brought a byte-compiler and the TclObj internal structure that maintained “native” values for types (I.e: a value “1.2” used as a floating-point value had its value cached as a float/double, versus reconstituting it from the string “1.2” every time it was needed. Note too this conversion happened nearly 20 years ago[0]. These days there’s some excitement surrounding the byte-compiling targeting LLVM[1], so faster still...

[0] https://www.tcl.tk/software/tcltk/8.0.html

[1] https://wiki.tcl.tk/40985

petregaius7 years ago
Not really. I did some prime number crunching and it's the slowest of them all. Maybe for simple scripts working with strings, but otherwise I'd pick Lua any day. LuaJIT just runs circles around other dynamic languages, only compiled stuff is able to beat it.
davidwpetre7 years ago
Tcl - or Ruby, Python without NumPy (which is an extension) or most scripting languages are not the right tool for the job for number crunching.
gaiusdavidw7 years ago
Right - both Python and Tcl/Tk are intended to offload heavy processing to C functions that they stitch together with high level controls.
ptrott2017davidw7 years ago
As DavidW points out Out of the box Tcl (like most scripting languages) was never designed for numeric processing. If you want to do numerics and number crunching in Tcl use one of the extensions such as Vectcl or the Mathemtcl. Tcl's C APi makes writing extensions (or wrapping existing libs) pretty straight forward too.
blacksqrpetre7 years ago
Note that the toplevel link is to a page full of simple techniques for enhancing base Tcl with C, Fortran, Lua etc. code, suggesting that Tcl offers the shortest path from dynamic language to serious number crunching power when it is needed.
vancan1typjmlp7 years ago
I personally think that might depend on what you mean by "slow". Tcl/Tk programs can be very light and memory-efficient -- you can pack a lot of features into a program that only takes 10 megabytes or less at runtime and that starts up and responds to input near-instantly. The event loop abstraction which is well-integrated in Tcl makes it easy to handle sockets, files, timers, and gui events all on one loop which done right can keep your program very responsive without needing to worry about multithreading.

In terms of pure runtime performance (CPU efficiency I guess), do you know of any reasonable benchmarks comparing TCL to say, Python and Ruby? I wonder if they might all be in the same ballpark?

pjmlpvancan1ty7 years ago
We were a startup with our own webstack, similar to AOLServer.

It was a great experience and we had quite a few in-house tooling that I only saw years later in Rails.

Those issues lead us to become early .NET adopters as Microsoft was searching partners for getting their products ready to ship when .NET 1.0 would be announced to the world.

Since then I only use scripting languages for shell scripting.

monetuspjmlp7 years ago
Isn't TCL almost as fast as C for certain things, like strings, but slow in other ways, like recursion? I haven't tried to do any performance optimizations yet really.
pjmlpmonetus7 years ago
Not when you are trying to build a server on top of it.
ridruejopjmlp7 years ago
It can be fairly fast for many use cases (specially because of async processing) and it was the basis one of the original scalable web servers that implemented multi-threading https://en.wikipedia.org/wiki/AOLserver
pjmlpridruejo7 years ago
AOLServer was the one we were measuring ourselves against, see other responses.
blacksqrpjmlp7 years ago
You had a web server faster than AOLServer, and you went out of business.

Hm.

pjmlp blacksqr7 years ago
We did not went out of business, I have not written that anywhere.

In fact, the .NET re-rewrite was quite successful and many of the people involved on that team went on to create a very successful product based on the field lessons we had with the Tcl version.

blacksqrpjmlp7 years ago
Time to consider that your experience may be outdated. Wub, to take one example, is a pure-Tcl web server whose performance is comparable to Apache. Network latency in handling socket I/O swamps any differences in code execution speed.
pjmlp blacksqr7 years ago
I doubt very much it could compete with our Apache/IIS plugin module, which had quite a few architecture layers moved from Tcl into C throughout its life.

And in the end we gave up for .NET.

ridruejodavidw7 years ago
Yes, the C API is one of the main reasons it got the ACM award back 1997 https://en.wikipedia.org/wiki/ACM_Software_System_Award
neulanddavidw7 years ago
How does Tcl's C API compare to Lua's? I've used that in a couple C projects and had a nice experience.
Timothycquinndavidw7 years ago
I'm not a C programmer but I spent some time analysing major parts of C code for a SAP integration library that I maintained to get a better understanding of the SAP RFC interfaces. Never had an issue with the C code and TCL side of the integrations very easy to reason and maintain whilst covering massive amounts of integrations to SAP.

More often then not, the problems I was involved with resolving on emergency escalation (line down) calls was on the SAP side of the world and never in the integration code in TCL and C.

The biggest drawback with the TCL implementation from our application vendor was that they did not implement multi threading for TCL and instead implemented a node.js like event loop. As SAP calls were locking, this did cause locks for all other TCL calls.

bchTimothycquinn7 years ago
> implemented a node.js like event loop

Who’s event-loop is like who’s? ;)

niftich7 years ago
Tcl is one of those rock-solid things that have been around for a while, where extrapolating from other projects of the same age you'd expect to find a somewhat muted community, and that's not the case at all.

Dormancy and stagnation can have many causes: author hiatus, the code clearing some threshold feature-complete, obsolescence by competing tech that solves similar problems. And it seems Tcl has managed to stay relevant through a combination of usefulness (like its standard lib, or Tk) sufficient paradigm difference to tech that came later, in some cases future tech building on top of it instead of replacing it wholesale, willingness to fix common gripes in the core (numbers, unicode, binary), and a developer-focused (instead of consumer-focused) approach to packaging. It's probably helped by being a captive scripting language on some embedded platforms too.

Languages that became big and popular in later years faced demands around the distribution of third-party packages. Perl became an early pioneer with a community repository, but later languages had to cope with the expectation of using a language-specific package manager that would download from the official repository, and automagically make packages available globally. Tcl encouraged user contributions in a less formal way [1] -- develop programs (code that uses builtin commands) or extensions (code that defines new commands) for your own use, and make them available if you so wish. This minor difference in attitude creates an ecosystem that feels different from a language package repo, where thousands of packages of varying quality sit, the long tail of them effectively unvetted.

[1] http://wiki.tcl.tk/2352

a_liebniftich7 years ago
Obviously this more stable, slow-and-steady approach to the package ecosystem has its pros and cons, but the best fruit of it was SQLite, possibly the best library ever made. D. Richard Hipp describes it as "a Tcl extension that escaped into the wild." [1]

[1] https://www.tcl.tk/community/tcl2017/assets/talk93/Paper.html

eridiusniftich7 years ago
Does Tcl actually have any sort of package manager? I've started writing Tcl scripts again lately after a decade of not touching the language, and and I'm thankful that so far macOS has seemed to include every extension I wanted to use in the Tcl it ships with, but I'm waiting the day I hit a missing extension and have no idea how to install it.
monetuseridius7 years ago
Activestate put teapot on github. I don't use it though, so I don't know how it works yet. > https://github.com/ActiveState/teapot

aside: Everything tcl just worked for you on macOS? Mac having 8.4 rather than 8.6 drove me nuts.

eridiusmonetus7 years ago
macOS has 8.5 (at least as of macOS 10.12), though that's still not 8.6. And I discovered the TclTLS package it ships with is old and doesn't even support TLSv1.2. But besides that it's been serviceable.
srean7 years ago
A couple of ideas that Tcl pioneered were:

(I) design for size, ease of extensibiliy and embedding, and (II) doing threads right.

(I) became a feature offered by many other languages, but I think Tcl was the one that popularized it. This crown belongs to Lua now.

(II) was not adopted as much as it should have been. I am looking at you Python. Interpreter as a library with interpreter state fully contained in an object, no globals or statics -- simplifies threads immensely. Run different interpreters in different threads, no need for global lock, no need for serialization-deserialization to talk to other interpreters in the same address space.

ridruejosrean7 years ago
For some time John Ousterhout was part of Sun Labs and Sun invested quite a bit of resources in it and bridges between Tcl and Java. The original idea was that Java was going to be the universal systems language and Tcl the universal scripting language. That was scrapped at some point because the messaging was too complex and it ended up being "Java for everything". Arguably we would have been better off with Java + Tcl than we ended up with Java + Javascript.

When that happened JO went to found Scriptics, but it never got the traction it should (not helped being branded as "parasites" by Stallman)

rwmjridruejo7 years ago
In one of those "could have been" moments, back in c.1994 I wrote a simple web browser which used SafeTcl (a sandboxed Tcl) to render. I got a few things wrong in hindsight - there was no static markup at all, the whole page was rendered by Tcl although there was a simple Tcl function to render blocks of text and TeX-like markup in Tcl strings.

Tcl is more pleasant than Javascript, but still has some horrible parts like poor support for structs, nothing really equivalent to a pointer, clumsy escaping, and (at the time) no support for dynamically linked extensions.

auxymrwmj7 years ago
FWIW, recent releases of Tcl have built-in dicts (hash tables), which greatly improve over the old "associative arrays".

What I missed the most when doing Tcl was support for first class functions / closures.

JimTCL seems to adress many of the pain points, for example it adds closures garbage-collected OO. http://jim.tcl.tk/index.html/doc/www/www/index.html

rurbanridruejo7 years ago
Why did RMS called them parasites?
ridruejorurban7 years ago
Scriptics was one of the earlier open-core commercial open source companies (before "open source" was coined) and Stallman took issue with it. He unleashed on JO during a panel for trying to make money from the language he had created with proprietary tools (Scriptics built and sold a proprietary IDE and debugger). There was a big debate at the time.
pwgrurban7 years ago
Why you should not use Tcl

https://wiki.tcl.tk/16730

DonHopkinspwg7 years ago
And with that diplomatically worded message, RMS kicked of The Infamous TCL War.

That was Stallman's response to Sun bombastically pushing TCL as the official scripting language of the web, BEFORE Live Oak / Java was a widely known (or evangelized) thing.

At the point anybody started talking about a Java/TCL bridge, it was already all over for TCL becoming the "ubiquitous scripting language of the Internet".

Sun's unilateral anointment of TCL as the official Internet scripting language trigged RMS's "Why you should not use Tcl" message, which triggered the TCL War, which triggered Sun to switch to Java.

After the TCL war finally subsided, Sun quietly pushed TCL aside and loudly evangelize Java instead. The TCL community was quite flustered and disappointed after first winning the title "ubiquitous scripting language of the Internet" and then having the title yanked away and given to Java.

Any talk of bridges were just table scraps for TCL, the redheaded bastard stepchild sitting outside on the back porch in the rain, smoking a cigarette and commiserating with NeWS and Self.

Tom Lord's description of what happened is insightful and accurate:

https://web.archive.org/web/20110102015130/http://basiscraft.com/0800-0100-the-tcl-war.html

>The Infamous Tcl War

>[...] Mr. Ousterhout had, a few years prior, developed Tcl while on the faculty of UC Berkeley - mainly, I think, to have a handy tool for other research and only secondarily as an experiment in language design. And he topped it off with Tk. Tcl/Tk took off in a huge way. It was easy to understand. The source code, written in Mr. Ousterhout's methodical and lucid style, was a joy to read. At the time, about the most convenient option for developing a GUI to run on a unix system was to write C code against the Motif toolkit - an ugly, expensive, and frequently disappointing process. With Tcl/Tk in hand, people started handing out new "mini-GUIs" for this and that, like candy. Tcl/Tk started to find application in some rather intense areas, like, for example, the "control station" software for some oil rigs. It was a smash hit.

>Meanwhile, I don't think I'm letting too many cats out of the bag here, the informal Silicon Valley social network of well placed hackers were quietly and unofficially circulating some very interesting confidential whitepapers from Sun Microsystems. One of their researchers, a fellow called Mr. Gosling, had dusted off a language he'd once led the design of called "Oak". Oak was originally intended for use in embedded systems. Its basic premise was that devices ought to be Turing complete and hackable, whenever possible. Oak's approach to statically verifiable byte-code comes from that origin. Mr. Gosling came out of Carnegie Mellon University and the attiude behind Oak was popular there. As one grad student had quipped a few years earlier: "If a light switch isn't Turing Complete I don't even want to touch it."

>In light of the rising star of web browsers, the folks at Sun conceived the notion of offering up a derivative of Oak to serve as the extension language for browsers. (It is probably worth mentioning here that Mr. Gosling was earlier well known for making one of the very first unix versions of Emacs.) Oak was re-named "Java" and the rest of its history is fairly well known.

>I've read, since then, that up to around that point Brendan Eich had been working on a Scheme-based extension language for Netscape Navigator. Such was the power of the hegemony of the high level folks at Sun that the word came down on Mr. Eich: "Get it done. And make it look like Java." Staying true to his sense of Self, he quickly knocked out the first implementation of Mocha, later renamed Javascript. This phenomenon of Sun's hegemony influencing other firms turns out to be a small pattern, as you'll see.

>Mr. Ousterhout was hired by Sun (later he would spin off a Tcl-centric start-up). The R&D; team there developed a vision:

>Java would be the heavy-lifting extension language for browsers. The earliest notions of the "browser as platform" and "browser as Microsoft-killer" date back to this time. Tcl, Sun announced, was to become the "ubiquitous scripting language of the Internet". Yes, they really pimped that vision for a while. And it was "the buzz" in the Valley. It was that pronouncement from the then-intimidating Sun that led to the Tcl wars.

>Mr. Eich, bless his soul, brute-forced passed them, abandoning Scheme and inventing Javascript. [...]

alfanickridruejo7 years ago
> Java was going to be the universal systems language

My heart stopped here for a while [0]. Are there any papers on Java as system language? I'd love to read more - as embedded developer, this really is out of my scope.

[0]: don't worry it does it 60 times per minute

ridruejoalfanick7 years ago
The genesis was actually as an embedded language :)

https://en.wikipedia.org/wiki/Java_(programming_language)#History

pjmlpalfanick7 years ago
Yes, Sun played with idea of putting a JVM on the Solaris kernel.

https://www.researchgate.net/publication/220938922_Writing_Solaris_device_drivers_in_Java

They also created SunSPOT before RasperryPI, Arduino were even ideas, where beyond a basic layer written in native code, everything else was Java based.

http://www.sunspotdev.org/

Also Java is actually used as systems language in some scenarios, just not on your desktop.

When it is compiled AOT to be deployed on bare-metal scenarios such like the SDKs being sold by PTC and Aicas (just two examples, there are other companies)

https://www.ptc.com/en/products/developer-tools/perc

https://www.aicas.com/cms/

When it is used on Android Things to write user space drivers.

https://developer.android.com/things/sdk/drivers/

DonHopkinspjmlp7 years ago
Even 16 or so years earlier in 1990, Sun played with the idea of putting a PostScript interpreter in the SunOS kernel.

Like NeWS was the Network extensible Window System, so NeFS was the Network extensible File System, or NFS 3.0.

It was actually a great idea, just a wee bit before its time, and very poorly named and positioned!

For example: If you want to make a copy of a file on the server, you can send a PostScript program that runs in the kernel and copies the file locally on the server in the kernel with ZERO context switches, instead of sending it over the net to the client, then back from the client to the server. Even if you rsh'ed the user command "cp" on the server, it would still incur context switching, but if your copy loop was running in the kernel then it didn't need to switch in and out and in and out for every block it copied.

There are more examples of why it's a great idea in the paper.

This comparison of NeWS to AJAX also applies NeFS, which is like kernel NeWS with file operations instead of a graphics library -- it also saves you lots of user/kernel context switches even if you're not doing any networking:

https://en.wikipedia.org/wiki/NeWS

>NeWS was architecturally similar to what is now called AJAX, except that NeWS coherently:

>- used PostScript code instead of JavaScript for programming.

>- used PostScript graphics instead of DHTML and CSS for rendering.

>- used PostScript data instead of XML and JSON for data representation.

It didn't go over very well because the unenlightened philistines of the time couldn't get their head around an API to the file system that wasn't compatible with creat open close read write and ioctl.

http://donhopkins.com/home/nfs3_0.pdf

Network Extensible File System Protocol Specification

1.0 Introduction

The Network Extensible File System protocol (NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome.

[...]

Example: Copy a File

Make a copy of file (foo) called (bar). Both files exist in the same directory dfh. The request starts by looking up the filehandle for the file to be copied and creates a filehandle for the copy. The loop operator executes a procedure that copies the file using 1K reads and writes. It maintains a running count of the number of bytes yet to be copied.

    % Copy a file
    %
    dfh (foo) lookup /foofh exch def % get filehandle for (foo)
    dfh (bar) create /barfh exch def % create filehandle for (bar)
    /bytes foofh getattr /fsize get def % get size of (foo) so we know how much to copy
    /offset 0 def % initialize offset for (bar)
    {
        /data foofh offset 1024 read def % read up to 1K from (foo)
        barfh offset data write % write up to 1K to (bar)
        /bytes bytes 1024 sub def % decrement byte count by 1024
        bytes 0 le { exit } if % if it’s < 0 then we’re done
        /offset offset 1024 add def % increment offset by 1024
    }
    loop
    barfh getattr 1 encodereply sendreply % return the attributes of the new file to client.
sreanDonHopkins7 years ago
Love HN for comments like these.
pjmlpDonHopkins7 years ago
Many thanks for the overview, NeWS had lots of great ideas, pity we got stuck with X11 on UNIX systems.
DonHopkinsDonHopkins7 years ago
Here's some old but interesting discussion about NeFS that I saved from Comp.protocols.NFS. (they were SHOCKED I say SHOCKED!!!)

http://donhopkins.com/home/archive/NeWS/Comp.protocols.nfs.NeFS-discussion.txt

    From: brent@terra.Sun.COM (Brent Callaghan)
    Newsgroups: comp.protocols.nfs
    Subject: NeFS Protocol Spec Available
    Keywords: NFS version 3
    Date: 13 Feb 90 01:57:31 GMT

    NeFS is not NFS.  It started out as an NFS protocol rev
    (NFS version 3) but its protocol is a radical departure
    from the NFS's remote procedure call model.  In order
    to avoid confusion with NFS we're calling it NeFS for
    now - Network Extensible File System (the similarity of
    the name to a window system protocol is entirely intentional).

    A draft spec for this new protocol is available via anonymous
    FTP from titan.rice.edu (128.42.1.30) as

        /public/nefs.doc.ps

    The file is ~250K of PostScript.  It prints 52 pages.
    The spec is not available in any other format.

    Comments, suggestions, flames etc are welcomed.  Direct them
    to "nfs3@sun.COM".

        Thanks

    Made in New Zealand -->  Brent Callaghan  @ Sun Microsystems
                 uucp: sun!bcallaghan
                 phone: (415) 336 1051


    From: mo@messy.bellcore.com (Michael O'Dell)
    Newsgroups: comp.protocols.nfs
    Subject: RE: NeFS protocol
    Date: 23 May 90 11:57:34 GMT
    Organization: Center for Chaotic Repeatabilty

    Dave Clark, Internet Architect, just gave a talk here at Bellcore and
    he was lobbying for exactly the kind of approach being proposed by the
    new NeFS protocol - don't send packets, send programs!

    I do recommend you read the NeFS document and think about it hard
    before you jump to any conclusions.  The proposed model is novel, and
    goes a long way toward moving us to "action at a distance" instead of
    "remoting local operations."  This turns out to be vitally important if
    networks are to scale with latency, since, as Dave Clark so eloquently
    showed, for large networks (ie, country-sized), as bandwidth goes to
    infinity latency goes to 30milliseconds, and you can't do anything
    about that, because you can't change the speed of light.  So, to do
    very well in the face of that, you must avoid round-trip delays like
    the plague.  Sending a program to the server can save many round-trips
    for many operations.

    Anyway, it is interesting reading.  Particularly in light of Dave
    Clark's talk.

        -Mike
tluyben2alfanick7 years ago
For embedded and systems language in actual use (a lot) today this is also relevant https://github.com/EnigmaBridge/javacard-curated-list/blob/master/README.md
bsderridruejo7 years ago
> When that happened JO went to found Scriptics, but it never got the traction it should (not helped being branded as "parasites" by Stallman)

The primary issue is that Tcl was a language designed on the wrong side of the memory cost divide. Many of the languages designed with "memory and CPU are expensive" got swept aside because newer languages that didn't make that assumption had a lot more features.

What is sad is that Tcl didn't STAY SMALL. In an era of 32K ROM/4K RAM microcontrollers, we don't really have a well-featured scripting language that works down there.

It also didn't help that Tcl went into a period of instability right as the Dot Com Boom/Crash went full tilt whereas Python, for example, was stable from 1997-2000 (A LOT of people standardized on 1.5 for a loong time)

And, as smart as he is, John Ousterhout was one of the big coding problems and getting him out of the codebase helped Tcl dramatically.

sedachvbsder7 years ago
> In an era of 32K ROM/4K RAM microcontrollers, we don't really have a well-featured scripting language that works down there.

What about Forth? http://www.forth.org/

bsdersedachv7 years ago
Just ... no. I'm trying to solve a problem, not play intellectual games for how to create stack threaded code.

I had really high hopes for Lua. But 1-based indexing when you're interfacing to C is just not flying. Apparently I'm not alone as Lua comes in at #9 on the "Dreaded" languages list:

https://insights.stackoverflow.com/survey/2017#technology-most-loved-dreaded-and-wanted-dreaded

sedachvbsder7 years ago
> 1-based indexing when you're interfacing to C is just not flying

That does not stop a lot of people. Not that it matters, because Lua will not run in 4KiB RAM.

Threaded code is a compiler/interpreter implementation technique. That makes as little sense as saying you won't use C, because you are "trying to solve a problem, not play intellectual games for how to create static single assignment code."

Instead of being an armchair programmer, you should actually try to learn those programming languages.

blacksqrbsder7 years ago
If small Tcl doesn't exist, it becomes necessary to invent it:

http://jim.tcl.tk/index.html/doc/www/www/index.html

KasianFranks7 years ago
ANNOUNCE: Digital Trader - java/tcl/tk web agent (1997) https://groups.google.com/forum/#!msg/comp.lang.tcl/5-KIZWGYLMI/fU-ZAIF8ZYMJ