Gå til innhold

infectedtech: The AAPEN Project


Anbefalte innlegg

AAPEN er kort for Adaptive Array of Power Efficient Nodes, og er ett prosjekt jeg holder paa med paa Murdoch University, School of Information Technology, som befinner seg I Perth, paa vestkysten av Australia.

 

Murdoch_CreamBox_4Col.jpg

 

aapen_logo.jpg

 

I god tradisjon tro tenkte jeg det var passende aa poste build-loggen av prosjektet her :)

 

IMGA0944-3.jpg

Many great components, an awesome primary node make :)

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL.

Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.

 

Kommentartraaden finner du her.

 

 

MvH,

[infected]

Endret av [Infected]
  • Liker 7
Lenke til kommentar
Videoannonse
Annonse

For enclosure, the Antec 4U22ATX has been chosen. This is a cheap yet solid rack mount case, with a reasonable price point, not to mention easily available without having to ship from the other coast.

 

 

 

Unboxing the Antec 4U22ATX, the 4 Rack Unit (RU) rack mountable housing destined for the primary node.

IMGA0950-3.jpg

 

The Antec in all its glory..

IMGA0956.jpg

 

The insides of the case.

IMGA0960-1.jpg

 

Antec has not skimped on the fans, Yate Loon make some great fans.

IMGA0962-1.jpg

 

Mainboard

 

Gigabyte 890FXA-UD7

IMGA0965-1.jpg

 

Unboxing..

IMGA0967.jpg

 

As it turns out, the 890FXA-UD7 is longer than standard full size ATX (its "XL-ATX" apparently, whatever that is), what a bummer!

IMGA0973-1.jpg

 

The board has to be exchanged for a more 'standard' ATX friendly one..

 

 

 

 

MvH

[infected]

Endret av [Infected]
  • Liker 3
Lenke til kommentar

Denne gangen er det lagring og komponenter til primaernoden som staar for tur.

 

 

 

The drive cage.

IMGA0975.JPG

 

Samsung TruDirect DVD writer.

IMGA0976.JPG

 

Quad 2.5" SATA drive backplane.

IMGA0977.JPG

 

2x PQI 64GB SandForce powered SSD's, one 32GB A-RAM SSD, and one Seagate 320GB XT hybrid drive..

The PQI's and A-RAM are all MLC's, not to mention the PQI's come with 64MB dram cache. The XT has a 4GB SLC read cache, and a nifty 32MB dram cache. This should make for some interesting comparisons ;)

IMGA0978.JPG

 

The A-RAM getting comfy with the drive bay.

IMGA0982.JPG

 

The MLC SSD trifecta.

IMGA0983.JPG

 

Each drives sits happily in its respective cage :)

IMGA0984.JPG

 

The 2nd drive cage will soon be filled with 7.2K rpm 500GB drives to be used for general purpose storage.

IMGA0986.JPG

 

The 1040SAS Quartet Mobile Backplane, and its assorted cables..

IMGA0988.JPG

 

Backplanes and DVD writer mounted.. FDD is missing for now.. damn, where did that black floppy drive go?

IMGA0991.JPG

 

A rather flashy photograph of how the drive bay looks..

IMGA0993.JPG

 

A nice pic of the rear end of the cage, too bad my camera handles flash so poorly.

IMGA0997.JPG

 

The ever so classic modular bay promotional picture!

IMGA0999.JPG

 

The AMD X6 1055T six-core processor.

IMGA0001.JPG

 

16 Gigabytes of DDR3 1333MHz memory, in a 4x4GB configuration.

IMGA0004.JPG

 

CUDA Accelerator

 

The EVGA GTS450 with 1GB of memory selected for this build.

IMGA0007.JPG

 

EVGA GTS450, the GF106 should clock in at about 600 gigaflops.

IMGA0009.JPG

 

 

 

More images to come :)

 

MvH

[infected]

Endret av [Infected]
  • Liker 4
Lenke til kommentar

The New Mainboard

 

 

 

Thanks to the great service at http://www.msy.com.au/ replacing the UD7 with the more ATX friendly UD5 was no problem :)

IMGA0012.JPG

 

Nothing much included, but a few SATA and IDE/floppy cables.

IMGA0014.JPG

 

The 890FXA-UD5 in all its glory

IMGA0015.JPG

 

The Powerplant

As power plant, the HX650 from Corsair has been selected.

IMGA0017.JPG

 

The contents of the box. This power supply is 95% modular, leaving only the 24pin ATX and the ATX12V connector.

IMGA0018.JPG

 

The ratings of the Corsair CMPSU-650HX

IMGA0023.JPG

 

The CPU Heatsink

The heat sink comes with some random thermal goop.

IMGA0026.JPG

 

Bobba Fett Original Space Gold gets rid of that with ease :)

IMGA0032.JPG

 

The CPU

The AMD Phenom II 1055T six-core processor in all its glory.

IMGA0033.JPG

 

Some quality thermal goop to choose from.. Arctic Silver 5 seem best suited for this job.

IMGA0035.JPG

 

An extremely flashy image of the 1055T mounted.

IMGA0036.JPG

 

The AS5 has been applied.

IMGA0040.JPG

 

CPU all sorted.

IMGA0042.JPG

 

Memory Bonanza

16 gigabytes of DDR3 1333MHz..

IMGA0044.JPG

 

The memory specs.

IMGA0048.JPG

 

All banks filled, and the board is maxed out.

IMGA0049.JPG

 

The EVGA GTS450 in all its glory.

IMGA0050.JPG

 

Mounted.

IMGA0051.JPG

 

 

 

 

More images to come :)

 

MvH

[infected]

Endret av [Infected]
  • Liker 4
Lenke til kommentar
  • 2 uker senere...

Tid for en aldri saa liten oppdatering.. saa mye aa gjore disse dager, lite tid til noen ting :(

 

 

 

A poorly lit rats nest!

IMGA0059.JPG

 

Most of the small wiring cleaned up

IMGA0064.JPG

 

Time for some poorly lit SATA cables!

IMGA0072.JPG

 

And even more cables..

IMGA0087.JPG

 

General Purpose Storage

The Western Digital 500GB 7.2k RPM drive with 16MB dram cache.

These four will be configured for RAID10 for both redundancy and performance :)

IMGA0096.JPG

 

The Black Edition in all its glory

IMGA0099.JPG

 

Ready for mounting..

IMGA0123.JPG

 

All mounted :)

IMGA0127.JPG

 

The top backplane is powered from the board's SATA controller

IMGA0133.JPG

 

Storage Controller

 

The Adaptec 1430SA PCIe 4 port SATA controller has finally arrived :)

IMGA0173.JPG

 

And mounted.

IMGA0179.JPG

 

The last few drives

Finalizing the storage for Polly.

 

Another Seagate Momentus XT 320GB hybrid drive.

IMGA0175.JPG

 

Two A-RAM 32GB Indilinx Barefoot powered SSD's

IMGA0182.JPG

 

SATA cables..

IMGA0181.JPG

 

Mounting the Barefoot's in the 2.5" to 3.5" bracket.

IMGA0190.JPG

 

Mounting the SSD's in the floppy bay.

IMGA0195.JPG

 

The drive cable salad..

IMGA0200.JPG

 

The LED lightshow of the Adaptec controller.

IMGA0219.JPG

 

Polly storage system:


  • 2x Seagate Momentus XT 320GB hybrid drives (RAID1)
    2x PQI 64GB SandForce SSD's (RAID0)
    2x A-RAM 32GB Barefoot SSD's (RAID0)
    4x Western Digital 500GB 7.2K Black Edition (RAID10)

 

IMGA0224.JPG

 

The drive bays in operation.

IMGA0230.JPG

 

The very feature-full (not to mention ultra-reliable) Gigabyte BIOS in all its glory :)

IMGA0238.JPG

 

 

 

Det var alt for denne gang, naa er det paa tide aa leke litt med RAID ;)

 

MvH

[infected]

Endret av [Infected]
  • Liker 3
Lenke til kommentar

Og for de der ute som liker tall :new_woot:

 

Seagate Momentus XT 320GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 284 MB in 3.01 seconds = 94.47 MB/sec

raid 1: Timing buffered disk reads: 204 MB in 3.01 seconds = 67.88 MB/sec

 

PQI S528 64GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 734 MB in 3.00 seconds = 244.47 MB/sec

raid 0: Timing buffered disk reads: 1100 MB in 3.00 seconds = 366.30 MB/sec

 

WD Black 500GB (Adaptec Serial ATA II RAID 1430SA):

single: Timing buffered disk reads: 276 MB in 3.01 seconds = 91.74 MB/sec

raid 10: Timing buffered disk reads: 464 MB in 3.01 seconds = 154.17 MB/sec

 

A-RAM Pro 32GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 552 MB in 3.01 seconds = 183.57 MB/sec

raid 0: Timing buffered disk reads: 548 MB in 3.00 seconds = 182.42 MB/sec

 

(mdraid 2.6.9-x86_64, Slackware 13.1)

 

The A-RAM's raid 0 performance makes me lol :)

 

MvH

[infected]

Endret av [Infected]
  • Liker 2
Lenke til kommentar
  • 2 uker senere...

Primaernoden er stort sett klargjort, paa tide aa sette sammen Node Array'et :)

 

 

Hardware

First the obligatory hardware shot!

 

IMGA0369.JPG

 


  • * 8x J&W Minix 785G AM3 Micro-ITX
    * 32GB DDR2 800MHz memory (4GB per. node)
    * 2x Corsair TX650 Power Supply Units
    * D-Link 24 port Gigabit switch
    * Various models of AMD processing units

 

 

 

Preparations

 

All nodes have been prepped and ready.

IMGA0375.JPG

 

Giving the nodes a full round of memtest.

IMGA0377.JPG

 

Constructing the Stacks

The nodes will be configured in two arrays of four nodes.

 

The mounting system is constructed from stainless steel rods cut to appropriate length, coupled with pvc reticulation tubing from Bunnings Warehouse ;)

IMGA0394.JPG

 

Node number one.

IMGA0396.JPG

 

Crossbars improve the structural integrity.

IMGA0399.JPG

 

The node stack.

IMGA0402.JPG

 

The first stack contain: 2x AMD 600e quad-cores and 2x AMD dual-core processing units.

IMGA0408.JPG

 

Glorious 16 Gigabytes of memory :D

IMGA0409.JPG

 

Constructing the secondary stack.

IMGA0435.JPG

 

Secondary Node Array completed :)

IMGA0439.JPG

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Det var alt for denne gang, flere bilder snart :)

 

MvH

[infected]

Endret av [Infected]
  • Liker 5
Lenke til kommentar
  • 3 uker senere...

Preparing the Enclosure

 

 

 

Stripping the Antec 4U22ATX case..

IMGA0877.JPG

 

What is the first thing to do when you receive your brand new rack mount case?

Thats right, you drill it to shits!

IMGA0880.JPG

 

Sometimes the drill just wont 'cut it', time to bring out the heavier guns.

IMGA0884.JPG

 

Brand new rack mount case? Not anymore!

IMGA0888.JPG

 

The AAPEN Electrical System

Constructing the ATX power plugs by hand.

IMGA0157.JPG

 

Just a few of all high-current connection blocks required by the project..

IMGA0249.JPG

 

Eventually the ATX plugs are coming together.

IMGA0242.JPG

 

Over 250 meters of assorted high-current power cable..

IMGA0448.JPG

 

Modifying the Power Supply

Two Corsair TX650 power supply units.

IMGA0352.JPG

 

Unboxing.

IMGA0353.JPG

 

The TX650 unboxed.

IMGA0483.JPG

 

Warranty void if removed.

IMGA0484.JPG

 

If you love high end hardware, you should probably look away now.

IMGA0464.JPG

 

Peek inside the Corsair (CWT) TX650 power supply unit.

IMGA0489.JPG

 

Sleeved cables are just so "2009" :)

IMGA0491.JPG

 

The PSU all prepped.

IMGA0467.JPG

 

The Node Arrays..

IMGA0442.JPG

 

The infectedtech AAPEN nodes.

IMGA0446.JPG

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Link til Kommentartraaden :)

 

 

MvH

[infected]

Endret av [Infected]
  • Liker 3
Lenke til kommentar
  • 2 uker senere...

Tid for en aldri saa liten update igjen, det som staar for tur naa er det sentrale strom fordelings-systemet i AAPEN, forovrig kan det nevnes at AAPEN er open-source, all programvare er lisensiert under GPL Version 3, hardware, design og hw-spesifikasjoner er lisensiert under TAPR NCL (ala. GPL for HW) og kan kopieres og benyttes fritt til alt ikke kommersielt bruk.

 

 

A package from one of my very favourite electronics vendors, Altronics.

IMGA0392cns.JPG

 

 

 

Lots and lots of relays. These relays are designed specifically for use in high-current high-temperature industrial installations.

IMGA0269.JPG

 

And yet another batch of relays.. earlier tested types and models etc.

IMGA0277.JPG

 

Constructing the AAPEN relay clusters.

IMGA0279.JPG

 

Wiring up the relay clusters.

IMGA0285.JPG

 

The solder used on the relays is certainly not RoHS compliant ;) Still, solder with a high lead content makes better joints with the copper conductor, and this is very important considering the very high currents in the system. Just got to make sure to wash hands afterwards ;)

IMGA0307.JPG

 

The wires used are 32x 0.2mm (17awg) and rated for use in high-current applications.

IMGA0309.JPG

 

Heat shrink, this stuff requires a very high tempearture to shrink, using the butane iron's furnace is great for this.

IMGA0312.JPG

 

And more wires. In total about 64 meters of high-current wiring, alongside 10-15 meters of low current wire is required for each cluster.

IMGA0317.JPG

 

The completed relay cluster.

IMGA0322.JPG

 

Taking the AAPEN Relay Clusters for a test run. Looks good :)

IMGA0288.JPG

 

Bench testing to ensure the currents and voltages experienced in the system is within the ATX 2.2 specifications.

(Actually, all AAPEN components and wiring is rated at very minimum twice that of the ATX2.2 specs... :))

IMGA0334.JPG

 

The voltages look really good as well :)

IMGA0325.JPG

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Det var alt for naa, vil ogsaa minne om kommentartraaden.

 

MvH

[infected]

Endret av [Infected]
  • Liker 2
Lenke til kommentar
  • 3 uker senere...

Blir bare en liten update denne gangen, paa foresporsler blir denne og neste

update av det mer tekniske slaget, der jeg vil forklare litt mer bagrunnen i hvordan

AAPEN virker og hva jeg har tenkt ved utviklingen av plattformen.

 

Denne updaten er dedikert til programvaren som kjores paa AAPEN, programvaren er skrevet

i C og Perl, der C er benyttet til de underliggende funksjonslaget, mens Perl er brukt til

selve interfacet i AAPEN. AAPEN vil bli sluppet i en Linux distro (kalt AAPEN Linux) basert

paa Slackware Linux, selv om programvaren i seg selv kan kjores paa de aller fleste POSIX

plattformer (*nix) med stotte for de nodvendige Perl modulene, samnt stotte for ANSI C bibliotekene.

 

 

 

 

cli_20.jpg

 

cli_21.jpg

 

cli_22.jpg

 

cli_23.jpg

 

cli_24.jpg

 

cli_25.jpg

 

 

 

 

MvH

[infected]

  • Liker 3
Lenke til kommentar
  • 6 måneder senere...
  • 3 uker senere...

Lang tid med lite updates her, men naa som semesteret endelig har startet opp igjen etter ferien

er det kanskje paa tide med en aldri saa liten update fra denne kanten ogsaa :)

 

 

 

 

The stacks..

IMGA0501-1.jpg

 

Preparing the enclosure

IMGA0502-1.jpg

 

Test fit

IMGA0503.jpg

 

Top view

IMGA0421.jpg

 

Front view

IMGA0428-1.jpg

 

Lots of wires

IMGA0430.jpg

 

Attaching the quick connectors

IMGA0508.jpg

 

Cheers!

IMGA0512.jpg

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

Endret av [Infected]
  • Liker 2
Lenke til kommentar
  • 2 uker senere...

Tid for enda en aldri saa liten oppdatering..

 

 

 

The 4RU enclosure, prepped and ready.

IMGA0513.jpg

 

Mounting the Relay control board, the board used is from Denkovi Electronics.

IMGA0515.jpg

 

Wiring up.

IMGA0516.jpg

 

The high current industrial realy cluster. there will be a considerable amount of Amperes passing thru here..

IMGA0523.jpg

 

High current wiring. everything is rated for the load, and then some.

IMGA0522.jpg

 

Sampling the layout.

IMGA0517.jpg

 

So many wires.. where will they all go? I forget.

IMGA0526-1.jpg

 

Top view of the relay cluster.

IMGA0536-2.jpg

 

Wiring up the control board.

IMGA0529.jpg

 

Selfless self promotion. :cool:

IMGA0534-2.jpg

 

All work and no beer, well is no way to work at all is it. Cheers!

IMGA0531-1.jpg

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Kommentartraaden

Endret av [Infected]
Lenke til kommentar
  • 2 uker senere...
  • 2 uker senere...

En aldri saa liten oppdatering, denne gangen er det balanserings-systemet som staar for tur, dette sorger for aa holde spenningene i systemet stabile, selv under en variabel last.

 

 

 

The AAPEN Power Balancing system, also known as the AAPB. Special thanks to Khaled Daabaj, phd computer engineering, for helping me with the math on this one.

 

The capacitors..

IMGA0541-1.jpg

 

Wiring up

IMGA0544.jpg

 

Heatshrinking

IMGA0547.jpg

 

The completed product.

IMGA0551.jpg

 

Mounted

IMGA0554.jpg

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

 

MvH

[infected]

  • Liker 1
Lenke til kommentar
  • 1 måned senere...

En aldri saa liten update, denne gangen med fokus paa testing :)

 

 

 

 

Inspection Guidelines..

IMGA0524.jpg

 

Datablad er alltid morsomt

IMGA0526-2.jpg

 

Ser fint ut

IMGA0529-1.jpg

 

Fint, fint

IMGA0530.jpg

 

Varmt

IMGA0531-2.jpg

 

Fint

IMGA0532-2.jpg

 

IB kontrolleren markerer som den varmeste delen av noden

IMGA0535-1.jpg

 

Stromfordelingssystemet varmeste delen av arrayet

IMGA0539-1.jpg

 

Point of reference

IMGA0543.jpg

 

 

 

 

og helt til slutt, en liten video tatt under testen :)

http://www.youtube.com/watch?v=27rT1P35sTw

 

Mvh

[infected]

Endret av [Infected]
Lenke til kommentar
  • 1 måned senere...
  • 3 uker senere...

En liten update denne gangen, for det meste random bilder tatt under forbedredelse for ny kabling samt installasjon av IB etc. :)

 

 

 

 

Campbeltown Loch, only good for killing braincells and degreasing CPU's.

IMGA0252-1.jpg

 

Diverse maalinger for kanalene kan tas opp for enda mer testing :!:

IMGA0276-1.jpg

 

Stromsystemet og strombalanseringssystemet

IMGA0277-2.jpg

 

Bilde foran av nodene for modifiseringer av stack'ene

IMGA0284.jpg

 

En Node Stack (stack #2) slik de ser ut for modifiseringer.. (1.0a)

IMGA0285-1.jpg

 

Nevermind har ogsaa faatt en aldri saa liten oppgradering. Fra NAS til NAS + SAN :w00t:

Adaptec SAS 31605 saa naa blir visst filserveren med i AAPEN ogsaa :yes:

IMGA0301.jpg

 

Oppgradering til infiniband! tonnevis med bilder kommer :w00t:

IMGA0227-1.jpg

 

Er slutten av enda ett semester naa (AAPEN has reached ALPHA!), etter siste innspurt med dokumenter og last-minute edits seff, saa naa god ferie med tid til aa jobbe litt mer med koden, samt kanskje poste noen flere updates ogsaa :!:

 

Og helt til slutt, tro det eller ei, AAPEN har naadd ALPHA!! :eek:

alpha_rc1_00.jpg

 

 

 

 

Har ogsaa noen heller spennende ting linet upp paa hardware fronten for semesteret som kommer, saa om du ikke har noe bedre ting aa ta deg til, folg med videre! :)

 

Forslag eller tips? Kommentartraaden

 

Mvh

[infected]

Endret av [Infected]
Lenke til kommentar
  • 2 uker senere...

Oppgradere nodene til infiniband, part 1;

 

Forste som trengs er ett par PCIe ribbon kabler

IMGA0293-1.jpg

 

En liten haug med IB HCA'er

IMGA0240-1.jpg

 

Preparere HCA'ene

IMGA0341-1.jpg

 

Prepare nodene

IMGA0336-2.jpg

 

Litt prototyping :hmm:

IMGA0354-1.jpg

 

Litt forstyrrelser :)

IMGA0349.jpg

 

Og demontering..

IMGA0355.jpg

 

Stay tuned!

IMGA0360-1.jpg

 

Mvh

[infected]

Endret av [Infected]
  • Liker 1
Lenke til kommentar
  • 2 uker senere...

Oppgradere nodene til infiniband, part 2;

 

 

Begynne aa ta ting fra hverandre

IMGA0361-1.jpg

 

IB kontrollerene klar til montering

IMGA0364.jpg

 

Tilpasse IB kablene, dette kommer til aa bli trangt

IMGA0365.jpg

 

Annen vinkel

IMGA0366-1.jpg

 

Testing, testing og mere testing. En skikkelig testplan, er coupe de grace, ivertfall i folge min supervisor :)

IMGA0372-1.jpg

 

Forste node i stack'en preparert, klargjore neste

IMGA0373-1.jpg

 

Ett detaljskudd :)

IMGA0376-1.jpg

 

Ikke helt enkelt aa faa bilde av IB kontrolleren gjemt bort der

IMGA0379-1.jpg

 

Bilde fra toppen, detaljarbeid dette

IMGA0384-1.jpg

 

Bilde fra front

IMGA0386-1.jpg

 

Test, test og mere test

IMGA0389.jpg

 

Saa de obligatoriske timene med bench og burn testing, alt maa selvfolgelig logges :w00t:

IMGA0396.jpg

 

Mvh

[infected]

Lenke til kommentar

Opprett en konto eller logg inn for å kommentere

Du må være et medlem for å kunne skrive en kommentar

Opprett konto

Det er enkelt å melde seg inn for å starte en ny konto!

Start en konto

Logg inn

Har du allerede en konto? Logg inn her.

Logg inn nå
  • Hvem er aktive   0 medlemmer

    • Ingen innloggede medlemmer aktive
×
×
  • Opprett ny...