In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, etc.), arranged so that the output of each element is the input of the next. Usually some amount of buffering is provided between consecutive elements. The information that flows in these pipelines is often a stream of records, bytes or bits.
The concept is also called the pipes and filters design pattern. It was named by analogy to a physical pipeline.
Multiprocessed pipelines:- Pipelines are often implemented in a multitasking OS, by launching all elements at the same time as processes, and automatically servicing the data read requests by each process with the data written by the upstream process. In this way, the CPU will be naturally switched among the processes by the scheduler so as to minimize its idle time. In other common models, elements are implemented as lightweight threads or as coroutines to reduce the OS overhead often involved with processes. Depending upon the OS, threads may be scheduled directly by the OS or by a thread manager. Coroutines are always scheduled by a coroutine manager of some form.
Usually, read and write requests are blocking operations, which means that the execution of the source process, upon writing, is suspended until all data could be written to the destination process, and, likewise, the execution of the destination process, upon reading, is suspended until at least some of the requested data could be obtained from the source process. Obviously, this cannot lead to a deadlock, where both processes would wait indefinitely for each other to respond, since at least one of the two processes will soon thereafter have its request serviced by the operating system, and continue to run.
For performance, most operating systems implementing pipes use pipe buffers, which allow the source process to provide more data than the destination process is currently able or willing to receive. Under most Unices and Unix-like operating systems, a special command is also available which implements a pipe buffer of potentially much larger and configurable size, typically called "buffer". This command can be useful if the destination process is significantly slower than the source process, but it is anyway desired that the source process can complete its task as soon as possible. E.g., if the source process consists of a command which reads an audio track from a CD and the destination process consists of a command which compresses the waveform audio data to a format like MP3. In this case, buffering the entire track in a pipe buffer would allow the CD drive to spin down more quickly, and enable the user to remove the CD from the drive before the encoding process has finished.
Such a buffer command can be implemented using system calls for reading and writing data. Wasteful busy waiting can be avoided by using facilities such as poll or select or multithreading.
CMS Pipelines is a port of the pipeline idea to VM/CMS and MVS systems. It supports much more complex pipeline structures than Unix shells, with steps taking multiple input streams and producing multiple output streams. (Such functionality is supported by the Unix kernel, but few programs use it as it makes for complicated syntax and blocking modes, although some shells do support it via arbitrary file descriptor assignment). Due to the different nature of IBM mainframe operating systems, it implements many steps inside CMS Pipelines which in Unix are separate external programs, but can also call separate external programs for their functionality. Also, due to the record-oriented nature of files on IBM mainframes, pipelines operate in a record-oriented, rather than stream-oriented manner.
On single-tasking operating systems, the processes of a pipeline have to be executed one by one in sequential order; thus the output of each process must be saved to a temporary file, which is then read by the next process. Since there is no parallelism or CPU switching, this version is called a "pseudo-pipeline".
For example, the command line interpreter of MS-DOS ('COMMAND.COM') provides pseudo-pipelines with a syntax superficially similar to that of Unix pipelines. The command "dir | sort | more" would have been executed like this (albeit with more complicated temporary file names):
All temporary files are stored in the directory pointed to by %TEMP%, or the current directory if %TEMP% isn't set.
Thus, pseudo-pipes acted like true pipes with a pipe buffer of unlimited size (disk space limitations notwithstanding), with the significant restriction that a receiving process could not read any data from the pipe buffer until the sending process finished completely. Besides causing disk traffic, if one doesn't install a harddisk cache such as SMARTDRV, that would have been unnecessary under multi-tasking operating systems, this implementation also made pipes unsuitable for applications requiring real-time response, like, for example, interactive purposes (where the user enters commands that the first process in the pipeline receives via stdin, and the last process in the pipeline presents its output to the user via stdout).
Also, commands that produce a potentially infinite amount of output, such as the yes command, cannot be used in a pseudo-pipeline, since they would run until the temporary disk space is exhausted, so the following processes in the pipeline could not even start to run.
Beside byte stream-based pipelines, there are also object pipelines. In an object pipeline, the processes output objects instead of texts; therefore removing the string parsing tasks that are common in UNIX shell scripts. Windows PowerShell uses this scheme and transfers .NET objects. Channels, found in the Limbo programming language, and the IPython ipipe extension are other examples of this metaphor.
Graphical environments such as RISC OS and ROX Desktop also make use of pipelines. Rather than providing a save dialog box containing a file manager to let the user specify where a program should write data, RISC OS and ROX provide a save dialog box containing an icon (and a field to specify the name). The destination is specified by dragging and dropping the icon. The user can drop the icon anywhere an already-saved file could be dropped, including onto icons of other programs. If the icon is dropped onto a program's icon, it's loaded and the contents that would otherwise have been saved are passed in on the new program's standard input stream.
For instance, a user browsing the world-wide web might come across a .gz compressed image which they want to edit and re-upload. Using GUI pipelines, they could drag the link to their de-archiving program, drag the icon representing the extracted contents to their image editor, edit it, open the save as dialog, and drag its icon to their uploading software.
Conceptually, this method could be used with a conventional save dialog box, but this would require the user's programs to have an obvious and easily accessible location in the filesystem that can be navigated to. In practice, this is often not the case, so GUI pipelines are rare.
The name 'pipeline' comes from a rough analogy with physical plumbing in that a pipeline usually allows information to flow in only one direction, like water often flows in a pipe.
Pipes and filters can be viewed as a form of functional programming, using byte streams as data objects; more specifically, they can be seen as a particular form of monad for I/O.
The concept of pipeline is also central to the Cocoon web development framework or to any XProc (the W3C Standards) implementations, where it allows a source stream to be modified before eventual display.
This pattern encourages the use of text streams as the input and output of programs. This reliance on text has to be accounted when creating graphic shells to text programs.
Process pipelines were invented by Douglas McIlroy, one of the designers of the first Unix shells, and greatly contributed to the popularity of that operating system. It can be considered the first non-trivial instance of software componentry.
The idea was eventually ported to other operating systems, such as DOS, OS/2, Windows NT, BeOS, AmigaOS, MorphOS and Mac OS X (the last one being a UNIX OS).
HOT DESIGNS ▪ Premium designs ▪ Designs by country ▪ Designs by U.S. state ▪ Most popular designs ▪ Newest, last added designs ▪ Unique designs ▪ Cheap, budget designs ▪ Design super sale DESIGNS BY THEME ▪ Accounting, audit designs ▪ Adult, sex designs ▪ African designs ▪ American, U.S. designs ▪ Animals, birds, pets designs ▪ Agricultural, farming designs ▪ Architecture, building designs ▪ Army, navy, military designs ▪ Audio & video designs ▪ Automobiles, car designs ▪ Books, e-book designs ▪ Beauty salon, SPA designs ▪ Black, dark designs ▪ Business, corporate designs ▪ Charity, donation designs ▪ Cinema, movie, film designs ▪ Computer, hardware designs ▪ Celebrity, star fan designs ▪ Children, family designs ▪ Christmas, New Year's designs ▪ Green, St. Patrick designs ▪ Dating, matchmaking designs ▪ Design studio, creative designs ▪ Educational, student designs ▪ Electronics designs ▪ Entertainment, fun designs ▪ Fashion, wear designs ▪ Finance, financial designs ▪ Fishing & hunting designs ▪ Flowers, floral shop designs ▪ Food, nutrition designs ▪ Football, soccer designs ▪ Gambling, casino designs ▪ Games, gaming designs ▪ Gifts, gift designs ▪ Halloween, carnival designs ▪ Hotel, resort designs ▪ Industry, industrial designs ▪ Insurance, insurer designs ▪ Interior, furniture designs ▪ International designs ▪ Internet technology designs ▪ Jewelry, jewellery designs ▪ Job & employment designs ▪ Landscaping, garden designs ▪ Law, juridical, legal designs ▪ Love, romantic designs ▪ Marketing designs ▪ Media, radio, TV designs ▪ Medicine, health care designs ▪ Mortgage, loan designs ▪ Music, musical designs ▪ Night club, dancing designs ▪ Photography, photo designs ▪ Personal, individual designs ▪ Politics, political designs ▪ Real estate, realty designs ▪ Religious, church designs ▪ Restaurant, cafe designs ▪ Retirement, pension designs ▪ Science, scientific designs ▪ Sea, ocean, river designs ▪ Security, protection designs ▪ Social, cultural designs ▪ Spirit, meditational designs ▪ Software designs ▪ Sports, sporting designs ▪ Telecommunication designs ▪ Travel, vacation designs ▪ Transport, logistic designs ▪ Web hosting designs ▪ Wedding, marriage designs ▪ White, light designs E-COMMERCE DESIGNS ▪ Magento store designs ▪ OpenCart store designs ▪ PrestaShop store designs ▪ CRE Loaded store designs ▪ Jigoshop store designs ▪ VirtueMart store designs ▪ osCommerce store designs ▪ Zen Cart store designs CMS DESIGNS ▪ Flash CMS designs ▪ Joomla CMS designs ▪ Mambo CMS designs ▪ Drupal CMS designs ▪ WordPress blog designs ▪ Forum designs ▪ phpBB forum designs ▪ PHP-Nuke portal designs ANIMATED WEBSITE DESIGNS ▪ Flash CMS designs ▪ Silverlight animated designs ▪ Silverlight intro designs ▪ Flash animated designs ▪ Flash intro designs ▪ XML Flash designs ▪ Flash 8 animated designs ▪ Dynamic Flash designs ▪ Flash animated photo albums ▪ Dynamic Swish designs ▪ Swish animated designs ▪ jQuery animated designs WEBSITE DESIGNS ▪ WebMatrix Razor designs ▪ HTML 5 designs ▪ Web 2.0 designs ▪ 3-color variation designs ▪ 3D, three-dimensional designs ▪ Artwork, illustrated designs ▪ Clean, simple designs ▪ CSS based website designs ▪ Full design packages ▪ Full ready websites ▪ Portal designs ▪ Stretched, full screen designs ▪ Universal, neutral designs CORPORATE ID DESIGNS ▪ Corporate identity sets ▪ Logo layouts, logo designs ▪ Logotype sets, logo packs ▪ PowerPoint, PTT designs ▪ Facebook themes VIDEO, SOUND & MUSIC ▪ Video e-cards ▪ After Effects video intros ▪ Special video effects ▪ Music tracks, music loops ▪ Stock music bank GRAPHICS & CLIPART ▪ Pro clipart & illustrations, $19/year ▪ 5,000+ icons by subscription ▪ Icons, pictograms |
| Super Offers |
| Super Offers |
| Custom Logo Design $149 ▪ Web Programming ▪ ID Card Printing ▪ Best Web Hosting ▪ eCommerce Software ▪ Add Your Link |
| © 1996-2013 MAGIA Internet Studio ▪ About ▪ Portfolio ▪ Photo on Demand ▪ Hosting ▪ Advertise ▪ Sitemap ▪ Privacy ▪ Maria Online
|
